{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Logistic 回归——糖尿病患者分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据集只有一个文件（diabetes.csv）：Pima Indians Diabetes Dataset 包括根据医疗\n",
    "记录的比马印第安人 5 年内糖尿病的发病情况，这是一个两类分类问题。每个类的\n",
    "样本数目数量不均等。一共有  768  个样本，每个样本有 8 个输入变量和 1 个输出\n",
    "变量。缺失值通常用零值编码。\n",
    "\n",
    "分别调用缺省参数LogisticRegression、LogisticRegression + GridSearchCV以及LogisticRegressionCV进行参数调优。实际应用中LogisticRegression + GridSearchCV或LogisticRegressionCV任选一个即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "#竞赛的评价指标为logloss\n",
    "from sklearn.metrics import log_loss  \n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据 & 数据探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "data = pd.read_csv(dpath +\"diabetes.csv\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据基本信息\n",
    "样本数目、特征维数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768, 9)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 768 entries, 0 to 767\n",
      "Data columns (total 9 columns):\n",
      "Pregnancies                 768 non-null int64\n",
      "Glucose                     768 non-null int64\n",
      "BloodPressure               768 non-null int64\n",
      "SkinThickness               768 non-null int64\n",
      "Insulin                     768 non-null int64\n",
      "BMI                         768 non-null float64\n",
      "DiabetesPedigreeFunction    768 non-null float64\n",
      "Age                         768 non-null int64\n",
      "Outcome                     768 non-null int64\n",
      "dtypes: float64(2), int64(7)\n",
      "memory usage: 54.1 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pregnancies                 0\n",
       "Glucose                     0\n",
       "BloodPressure               0\n",
       "SkinThickness               0\n",
       "Insulin                     0\n",
       "BMI                         0\n",
       "DiabetesPedigreeFunction    0\n",
       "Age                         0\n",
       "Outcome                     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看是否有空值\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pregnancies     Glucose  BloodPressure  SkinThickness     Insulin  \\\n",
       "count   768.000000  768.000000     768.000000     768.000000  768.000000   \n",
       "mean      3.845052  120.894531      69.105469      20.536458   79.799479   \n",
       "std       3.369578   31.972618      19.355807      15.952218  115.244002   \n",
       "min       0.000000    0.000000       0.000000       0.000000    0.000000   \n",
       "25%       1.000000   99.000000      62.000000       0.000000    0.000000   \n",
       "50%       3.000000  117.000000      72.000000      23.000000   30.500000   \n",
       "75%       6.000000  140.250000      80.000000      32.000000  127.250000   \n",
       "max      17.000000  199.000000     122.000000      99.000000  846.000000   \n",
       "\n",
       "              BMI  DiabetesPedigreeFunction         Age     Outcome  \n",
       "count  768.000000                768.000000  768.000000  768.000000  \n",
       "mean    31.992578                  0.471876   33.240885    0.348958  \n",
       "std      7.884160                  0.331329   11.760232    0.476951  \n",
       "min      0.000000                  0.078000   21.000000    0.000000  \n",
       "25%     27.300000                  0.243750   24.000000    0.000000  \n",
       "50%     32.000000                  0.372500   29.000000    0.000000  \n",
       "75%     36.600000                  0.626250   41.000000    1.000000  \n",
       "max     67.100000                  2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 各属性的统计特性\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处得到各属性的样本数目、均值、标准差、最小值、1/4分位数（25%）、中位数（50%）、3/4分位数（75%）、最大值 可初步了解各特征的分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAE+JJREFUeJzt3X+0ZWV93/H3BwbUoHH4MbIow2SwjmnJUpGMFEOrEW0q2jJIxcSyZIKzOsmqURLS1NGkksTYaCJiqC3trGAYLIUgFRmUxkxHwBUTKDMoIKJlpARmYZkB+aUULPDtH+e5cBj23LtnhnPP4d73a62zzt7Pefa53+sa7se9n72fJ1WFJEk72mvcBUiSJpMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp04JxF7AnDjrooFq6dOm4y5Ck55XNmzffW1WLZur3vA6IpUuXsmnTpnGXIUnPK0n+tk8/LzFJkjoZEJKkTgaEJKmTASFJ6mRASJI6jTQgktyR5OYk30yyqbUdkGRDktva+/6tPUnOSbIlyU1JjhplbZKk6c3GGcSbqurIqlre9tcAG6tqGbCx7QMcDyxrr9XAubNQmyRpJ8ZxiWkFsK5trwNOHGq/oAauBRYmOWQM9UmSGH1AFPCXSTYnWd3aDq6q7wO095e19kOBu4aO3draJEljMOonqY+tqruTvAzYkOQ70/RNR1s9q9MgaFYDLFmyZI8L/NnfumCPv0Nzz+Y/PnXcJUhjN9IziKq6u71vAy4Djgbumbp01N63te5bgcOGDl8M3N3xnWuranlVLV+0aMapRCRJu2lkAZFkvyQvmdoGfgH4FrAeWNm6rQQub9vrgVPb3UzHAA9OXYqSJM2+UV5iOhi4LMnUz/mvVfUXSa4HLkmyCrgTOLn1vxJ4G7AFeAQ4bYS1SZJmMLKAqKrbgdd0tN8HvLmjvYD3jaoeSdKu8UlqSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdRh4QSfZO8o0kX2r7hye5LsltSf48yb6t/QVtf0v7fOmoa5Mk7dxsnEGcDtw6tP8J4OyqWgbcD6xq7auA+6vqFcDZrZ8kaUxGGhBJFgNvB/607Qc4Dri0dVkHnNi2V7R92udvbv0lSWMw6jOITwP/Bniy7R8IPFBVj7f9rcChbftQ4C6A9vmDrb8kaQxmDIgk+yXZq22/MskJSfbpcdw/BbZV1ebh5o6u1eOz4e9dnWRTkk3bt2+fqQxJ0m7qcwbxNeCFSQ4FNgKnAef3OO5Y4IQkdwAXM7i09GlgYZIFrc9i4O62vRU4DKB9/lLgBzt+aVWtrarlVbV80aJFPcqQJO2OPgGRqnoEOAn491X1DuCImQ6qqg9V1eKqWgr8EvDVqjoFuAp4Z+u2Eri8ba9v+7TPv1pVzzqDkCTNjl4BkeT1wCnAl1vbgmn6z+SDwBlJtjAYYzivtZ8HHNjazwDW7MHPkCTtoT5/6H8d+BBwWVXdkuTlDM4Cequqq4Gr2/btwNEdfR4FTt6V75Ukjc6MAVFV1wDXJNmv7d8OfGDUhUmSxqvPXUyvT/Jt2sNuSV6T5D+OvDJJ0lj1GYP4NPBPgPsAqupG4A2jLEqSNH69HpSrqrt2aHpiBLVIkiZIn0Hqu5L8HFBtYr0P8My5lSRJc1CfM4hfBd7HYCqMrcCRbV+SNIf1uYvpXgbPQEiS5pE+dzGtS7JwaH//JJ8dbVmSpHHrc4np1VX1wNROVd0PvHZ0JUmSJkGfgNgryf5TO0kOYM+m2pAkPQ/0+UN/FvDXSaYW+TkZ+NjoSpIkTYI+g9QXJNkMvInBmg0nVdW3R16ZJGms+l4q+g6D9aMXACRZUlV3jqwqSdLYzRgQSd4PnAncw+AJ6jBY6e3Voy1NkjROfc4gTgd+uqruG3UxkqTJ0ecupruAB0ddiCRpsvQ5g7gduDrJl4HHphqr6lMjq0qSNHZ9AuLO9tq3vSRJ80Cf21x/DyDJflX1o9GXJEmaBK4oJ0nq5IpykqROrignSerkinKSpE6uKCdJ6jTtGUSSvYH3VJUryknSPDPtGURVPQGsmKVaJEkTpM8YxNeTfAb4c+Cp5yCq6oaRVSVJGrs+AfFz7f33h9oKOO65L0eSNClmGoPYCzi3qi6ZpXokSRNipjGIJ4Ffm6VaJEkTpM9trhuS/OskhyU5YOo18sokSWPVZwzive19+NmHAl7+3JcjSZoUfWZzPXw2CpEkTZY+a1Kf2tVeVRfMcNwLga8BL2g/59KqOjPJ4cDFwAHADQwexPtxkhcAFwA/y2BiwF+sqjt24XeRJD2H+oxBvG7o9Y+A3wVO6HHcY8BxVfUaBtNzvDXJMcAngLOrahlwP7Cq9V8F3F9VrwDObv0kSWPS5xLT+4f3k7wU+FyP4wr4Ydvdp72mnp/4F619HYPAOZfBE9u/29ovBT6TJO17JEmzrNd03zt4BFjWp2OSvZN8E9gGbAC+BzxQVY+3LlsZTAJIe78LoH3+IHDgbtQnSXoO9BmDuILB//OHQaAcAfR6cK7N5XRkkoXAZcDf7+o29aOm+Wy4ntXAaoAlS5b0KUOStBv63Ob6yaHtx4G/raqtu/JDquqBJFcDxwALkyxoZwmLgbtbt63AYcDWJAuAlwI/6PiutcBagOXLl3v5SZJGpM8lpjuB66rqmqr6OnBfkqUzHZRkUTtzIMmLgLcwWGjoKuCdrdtK4PK2vb7t0z7/quMPkjQ+fQLi88CTQ/tPtLaZHAJcleQm4HpgQ1V9CfggcEaSLQzGGM5r/c8DDmztZwBr+v0KkqRR6HOJaUFV/Xhqpz2zsO9MB1XVTcBrO9pvB47uaH8UOLlHPZKkWdDnDGJ7kqeee0iyArh3dCVJkiZBnzOIXwUubIsGwWAwufPpaknS3NHnQbnvAcckeTGQqnp49GVJksZtxktMSf5dkoVV9cOqejjJ/kn+YDaKkySNT58xiOOr6oGpnaq6H3jb6EqSJE2CPgGxd5tpFXjqmYYXTNNfkjQH9Bmk/i/AxiR/xmDqi/cymGRPkjSH9Rmk/qP2sNtbWtNHq+oroy1LkjRufc4gAL7B09N1f2N05Uiacufvv2rcJWgCLfnIzbP2s/rcxfQu4H8ymB/pXcB1Sd45/VGSpOe7PmcQvw28rqq2wWASPuB/MFjUR5I0R/W5i2mvqXBo7ut5nCTpeazPGcRfJPkKcFHb/0XgytGVJEmaBH3uYvqtJCcB/5DBqm9rq+qykVcmSRqrXncxVdUXgC+MuBZJ0gRxLEGS1MmAkCR12mlAJNnY3j8xe+VIkibFdGMQhyR5I3BCkosZDFA/papuGGllkqSxmi4gPgKsARYDn9rhswKOG1VRkqTx22lAVNWlwKVJ/m1VfXQWa5IkTYA+z0F8NMkJwBta09VV9aXRliVJGrc+k/X9IXA68O32Or21SZLmsD4Pyr0dOLKqngRIso7BlN8fGmVhkqTx6vscxMKh7ZeOohBJ0mTpcwbxh8A3klzF4FbXN+DZgyTNeX0GqS9KcjXwOgYB8cGq+j+jLkySNF59J+v7PrB+xLVIkiaIczFJkjoZEJKkTtMGRJK9knxrtoqRJE2OaQOiPftwY5Ils1SPJGlC9LnEdAhwS5KNSdZPvWY6KMlhSa5KcmuSW5Kc3toPSLIhyW3tff/WniTnJNmS5KYkR+3ZryZJ2hN97mL6vd387seB36yqG5K8BNicZAPwy8DGqvp4kjUMZoz9IHA8sKy9/gFwbnuXJI3BjGcQVXUNcAewT9u+HphxLYiq+v7UmhFV9TBwK3AosAJY17qtA05s2yuAC2rgWmBhkkN27deRJD1X+kzW9y+BS4H/3JoOBb64Kz8kyVLgtcB1wMHtuYqp5yteNvS9dw0dtrW1SZLGoM8YxPuAY4GHAKrqNp7+oz6jJC8G/hvw61X10HRdO9qq4/tWJ9mUZNP27dv7liFJ2kV9AuKxqvrx1E6SBXT84e6SZB8G4XBhVX2hNd8zdemovW9r7VuBw4YOXwzcveN3VtXaqlpeVcsXLVrUpwxJ0m7oExDXJPkw8KIk/xj4PHDFTAclCXAecGtVDS9Zuh5Y2bZXApcPtZ/a7mY6Bnhw6lKUJGn29bmLaQ2wCrgZ+BXgSuBPexx3LPAe4OYk32xtHwY+DlySZBVwJ3By++xK4G3AFuAR4LSev4MkaQT6zOb6ZFsk6DoGl5a+W1UzXmKqqr+ie1wB4M0d/YvBeIckaQLMGBBJ3g78J+B7DP7gH57kV6rqv4+6OEnS+PS5xHQW8Kaq2gKQ5O8CXwYMCEmaw/oMUm+bCofmdp6+80iSNEft9AwiyUlt85YkVwKXMBiDOJnB09SSpDlsuktM/2xo+x7gjW17O7D/yCqSJE2EnQZEVXmbqSTNY33uYjoceD+wdLh/VZ0wurIkSePW5y6mLzJ4IvoK4MnRliNJmhR9AuLRqjpn5JVIkiZKn4D4kyRnAn8JPDbVOLXWgyRpbuoTEK9iMKfScTx9ianaviRpjuoTEO8AXj485bckae7r8yT1jcDCURciSZosfc4gDga+k+R6njkG4W2ukjSH9QmIM0dehSRp4vRZD+Ka2ShEkjRZ+jxJ/TBPr0G9L7AP8KOq+slRFiZJGq8+ZxAvGd5PciJw9MgqkiRNhD53MT1DVX0Rn4GQpDmvzyWmk4Z29wKW8/QlJ0nSHNXnLqbhdSEeB+4AVoykGknSxOgzBuG6EJI0D0235OhHpjmuquqjI6hHkjQhpjuD+FFH237AKuBAwICQpDlsuiVHz5raTvIS4HTgNOBi4KydHSdJmhumHYNIcgBwBnAKsA44qqrun43CJEnjNd0YxB8DJwFrgVdV1Q9nrSpJ0thN96DcbwJ/B/gd4O4kD7XXw0kemp3yJEnjMt0YxC4/ZS1JmjsMAUlSJwNCktTJgJAkdTIgJEmdRhYQST6bZFuSbw21HZBkQ5Lb2vv+rT1JzkmyJclNSY4aVV2SpH5GeQZxPvDWHdrWABurahmwse0DHA8sa6/VwLkjrEuS1MPIAqKqvgb8YIfmFQyeyKa9nzjUfkENXAssTHLIqGqTJM1stscgDq6q7wO095e19kOBu4b6bW1tz5JkdZJNSTZt3759pMVK0nw2KYPU6WjrXLWuqtZW1fKqWr5o0aIRlyVJ89dsB8Q9U5eO2vu21r4VOGyo32Lg7lmuTZI0ZLYDYj2wsm2vBC4faj+13c10DPDg1KUoSdJ49FmTerckuQj4eeCgJFuBM4GPA5ckWQXcCZzcul8JvA3YAjzCYN0JSdIYjSwgqurdO/nozR19C3jfqGqRJO26SRmkliRNGANCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnSYqIJK8Ncl3k2xJsmbc9UjSfDYxAZFkb+A/AMcDRwDvTnLEeKuSpPlrYgICOBrYUlW3V9WPgYuBFWOuSZLmrUkKiEOBu4b2t7Y2SdIYLBh3AUPS0VbP6pSsBla33R8m+e5Iq5pfDgLuHXcRkyCfXDnuEvRM/tuccmbXn8pd9lN9Ok1SQGwFDhvaXwzcvWOnqloLrJ2touaTJJuqavm465B25L/N8ZikS0zXA8uSHJ5kX+CXgPVjrkmS5q2JOYOoqseT/BrwFWBv4LNVdcuYy5KkeWtiAgKgqq4Erhx3HfOYl+40qfy3OQapetY4sCRJEzUGIUmaIAaEnOJEEyvJZ5NsS/KtcdcyHxkQ85xTnGjCnQ+8ddxFzFcGhJziRBOrqr4G/GDcdcxXBoSc4kRSJwNCvaY4kTT/GBDqNcWJpPnHgJBTnEjqZEDMc1X1ODA1xcmtwCVOcaJJkeQi4G+An06yNcmqcdc0n/gktSSpk2cQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaE5r0ki5NcnuS2JN9L8iftmZDpjvnwbNUnjYsBoXktSYAvAF+sqmXAK4EXAx+b4VADQnOeAaH57jjg0ar6M4CqegL4DeC9Sf5Vks9MdUzypSQ/n+TjwIuSfDPJhe2zU5PclOTGJJ9rbT+VZGNr35hkSWs/P8m5Sa5KcnuSN7Z1D25Ncv7Qz/uFJH+T5IYkn0/y4ln7X0XCgJB+Btg83FBVDwF3spM126tqDfB/q+rIqjolyc8Avw0cV1WvAU5vXT8DXFBVrwYuBM4Z+pr9GYTTbwBXAGe3Wl6V5MgkBwG/A7ylqo4CNgFnPBe/sNRX538A0jwSumev3Vl7l+OAS6vqXoCqmlq/4PXASW37c8AfDR1zRVVVkpuBe6rqZoAktwBLGUyaeATw9cFVMPZlMOWENGsMCM13twD/fLghyU8ymOH2QZ55lv3CnXxH3zAZ7vNYe39yaHtqfwHwBLChqt7d43ulkfASk+a7jcBPJDkVnlqC9SwGS13eDhyZZK8khzFYfW/K/0uyz9B3vCvJge07Dmjtf81gdlyAU4C/2oW6rgWOTfKK9p0/keSVu/rLSXvCgNC8VoPZKt8BnJzkNuB/AY8yuEvp68D/Bm4GPgncMHToWuCmJBe22W8/BlyT5EbgU63PB4DTktwEvIenxyb61LUd+GXgonb8tcDf293fU9odzuYqSerkGYQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE7/H6faJrOoDn8hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Outcome 分布，看看各类样本分布是否均衡\n",
    "sns.countplot(data.Outcome);\n",
    "pyplot.xlabel('Outcome');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡。交叉验证对分类任务缺省的是采用StratifiedKFold，在每折采样时根据各类样本按比例采样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练数据和测试数据分割\n",
    "（随机选择20%的数据作为测试集）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data = data.sample(frac=1)  # 对原始数据进行随机排序\n",
    "# test = data.head(154)   # 测试数据20%\n",
    "# train = data.tail(614)  # 训练数据80%\n",
    "\n",
    "# 从原始数据中分离输入特征x和输出y\n",
    "y = data['Outcome'].values\n",
    "X = data.drop('Outcome', axis = 1)\n",
    "\n",
    "#将数据分割训练数据与测试数据\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 随机采样20%的数据构建测试样本，其余作为训练样本\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=33, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据标准化\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 初始化特征的标准化器\n",
    "ss_X = StandardScaler()\n",
    "\n",
    "# 分别对训练和测试数据的特征进行标准化处理\n",
    "X_train = ss_X.fit_transform(X_train)\n",
    "X_test = ss_X.transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### default Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "lr= LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logloss of each fold is:  [0.46129644 0.46510588 0.56426612 0.44377717 0.46617809]\n",
      "cv logloss is: 0.48012474004701106\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python36\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# 交叉验证用于评估模型性能和进行参数调优（模型选择）\n",
    "#分类任务中交叉验证缺省是采用StratifiedKFold\n",
    "from sklearn.cross_validation import cross_val_score\n",
    "loss = cross_val_score(lr, X_train, y_train, cv=5, scoring='neg_log_loss')\n",
    "print ('logloss of each fold is: ', -loss)\n",
    "print ('cv logloss is:', -loss.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则化的 Logistic Regression及参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "logistic回归的需要调整超参数有：C（正则系数C = 1 / lambda，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） 目标函数为：J = sum(logloss(f(xi), yi)) + C* penalty\n",
    "\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同： 设置候选参数集合 调用GridSearchCV 调用fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
       "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
       "          verbose=0, warm_start=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'penalty': ['l1', 'l2'], 'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "tuned_parameters = dict(penalty = penaltys, C = Cs)  #GridSearchCV要求dictionary参数形式\n",
    "\n",
    "lr_penalty= LogisticRegression()\n",
    "grid= GridSearchCV(lr_penalty, tuned_parameters,cv=5, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.0004004 , 0.00080061, 0.00059981, 0.00060043, 0.00040021,\n",
       "        0.00080042, 0.00060096, 0.00080051, 0.00100045, 0.00080051,\n",
       "        0.00100355, 0.00100079, 0.00080042, 0.00080037]),\n",
       " 'mean_score_time': array([0.00100021, 0.00060048, 0.00060058, 0.00060105, 0.00060048,\n",
       "        0.00060048, 0.00060034, 0.00060034, 0.00080323, 0.00059748,\n",
       "        0.00059857, 0.00060024, 0.00060034, 0.00060067]),\n",
       " 'mean_test_score': array([-0.69314718, -0.64214833, -0.6721329 , -0.52844007, -0.48658747,\n",
       "        -0.47999943, -0.48043821, -0.48017599, -0.4809023 , -0.48086932,\n",
       "        -0.48095197, -0.48095123, -0.48095809, -0.48095956]),\n",
       " 'mean_train_score': array([-0.69314718, -0.6412946 , -0.67079105, -0.52380684, -0.47502573,\n",
       "        -0.46674403, -0.46228783, -0.46214818, -0.46206769, -0.46206619,\n",
       "        -0.46206531, -0.4620653 , -0.46206529, -0.46206529]),\n",
       " 'param_C': masked_array(data=[0.001, 0.001, 0.01, 0.01, 0.1, 0.1, 1, 1, 10, 10, 100,\n",
       "                    100, 1000, 1000],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_penalty': masked_array(data=['l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1', 'l2', 'l1',\n",
       "                    'l2', 'l1', 'l2', 'l1', 'l2'],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.001, 'penalty': 'l1'},\n",
       "  {'C': 0.001, 'penalty': 'l2'},\n",
       "  {'C': 0.01, 'penalty': 'l1'},\n",
       "  {'C': 0.01, 'penalty': 'l2'},\n",
       "  {'C': 0.1, 'penalty': 'l1'},\n",
       "  {'C': 0.1, 'penalty': 'l2'},\n",
       "  {'C': 1, 'penalty': 'l1'},\n",
       "  {'C': 1, 'penalty': 'l2'},\n",
       "  {'C': 10, 'penalty': 'l1'},\n",
       "  {'C': 10, 'penalty': 'l2'},\n",
       "  {'C': 100, 'penalty': 'l1'},\n",
       "  {'C': 100, 'penalty': 'l2'},\n",
       "  {'C': 1000, 'penalty': 'l1'},\n",
       "  {'C': 1000, 'penalty': 'l2'}],\n",
       " 'rank_test_score': array([14, 12, 13, 11, 10,  1,  3,  2,  5,  4,  7,  6,  8,  9]),\n",
       " 'split0_test_score': array([-0.69314718, -0.64371675, -0.66946739, -0.52816851, -0.48618075,\n",
       "        -0.4678555 , -0.46346352, -0.46129644, -0.46108725, -0.46088502,\n",
       "        -0.46086744, -0.4608487 , -0.46084952, -0.46084513]),\n",
       " 'split0_train_score': array([-0.69314718, -0.64085132, -0.66189542, -0.52529099, -0.4782405 ,\n",
       "        -0.47085398, -0.46684251, -0.46669378, -0.46662379, -0.46662226,\n",
       "        -0.46662151, -0.46662149, -0.46662149, -0.46662148]),\n",
       " 'split1_test_score': array([-0.69314718, -0.64113725, -0.67517494, -0.52518603, -0.47166813,\n",
       "        -0.47077815, -0.46426306, -0.46510588, -0.46463666, -0.46473286,\n",
       "        -0.46468657, -0.46469927, -0.46469015, -0.46469595]),\n",
       " 'split1_train_score': array([-0.69314718, -0.64188719, -0.67797602, -0.52532854, -0.4797048 ,\n",
       "        -0.47076994, -0.46692698, -0.46678157, -0.46671609, -0.4667146 ,\n",
       "        -0.4667139 , -0.46671388, -0.46671388, -0.46671388]),\n",
       " 'split2_test_score': array([-0.69314718, -0.64575466, -0.6680453 , -0.5512625 , -0.54752697,\n",
       "        -0.54301469, -0.56470004, -0.56426612, -0.56848081, -0.56834734,\n",
       "        -0.56879885, -0.56879096, -0.56883355, -0.5688357 ]),\n",
       " 'split2_train_score': array([-0.69314718, -0.63908575, -0.66162164, -0.5135644 , -0.45564498,\n",
       "        -0.44783203, -0.44194578, -0.44184498, -0.44173182, -0.44173047,\n",
       "        -0.44172922, -0.4417292 , -0.44172919, -0.44172919]),\n",
       " 'split3_test_score': array([-0.69314718, -0.63856226, -0.67581323, -0.50975335, -0.45503013,\n",
       "        -0.44706595, -0.44366085, -0.44377717, -0.44397794, -0.44400066,\n",
       "        -0.44402916, -0.4440332 , -0.44403567, -0.44403656]),\n",
       " 'split3_train_score': array([-0.69314718, -0.64333601, -0.67874616, -0.5309552 , -0.4838244 ,\n",
       "        -0.47509784, -0.47066671, -0.47052445, -0.47044461, -0.47044314,\n",
       "        -0.47044227, -0.47044226, -0.47044225, -0.47044225]),\n",
       " 'split4_test_score': array([-0.69314718, -0.64152376, -0.67221592, -0.52767402, -0.47216084,\n",
       "        -0.47104103, -0.46582378, -0.46617809, -0.46606916, -0.46612355,\n",
       "        -0.46612033, -0.4661268 , -0.46612427, -0.46612722]),\n",
       " 'split4_train_score': array([-0.69314718, -0.64131272, -0.67371601, -0.52389506, -0.47771395,\n",
       "        -0.46916638, -0.46505718, -0.46489609, -0.46482214, -0.46482048,\n",
       "        -0.46481965, -0.46481964, -0.46481963, -0.46481963]),\n",
       " 'std_fit_time': array([4.90388050e-04, 4.00304865e-04, 4.89745942e-04, 4.90253176e-04,\n",
       "        4.90154588e-04, 4.00209697e-04, 4.90681182e-04, 4.00257253e-04,\n",
       "        3.16297988e-07, 4.00257196e-04, 5.72085368e-06, 2.43140197e-07,\n",
       "        4.00209555e-04, 4.00185596e-04]),\n",
       " 'std_score_time': array([5.51978917e-07, 4.90290765e-04, 4.90368678e-04, 4.90758616e-04,\n",
       "        4.90290718e-04, 4.90290904e-04, 4.90173941e-04, 4.90173918e-04,\n",
       "        4.01647555e-04, 4.87867008e-04, 4.88744656e-04, 4.90096043e-04,\n",
       "        4.90174474e-04, 4.90446445e-04]),\n",
       " 'std_test_score': array([0.        , 0.00243715, 0.00305426, 0.0132657 , 0.03205912,\n",
       "        0.03276814, 0.04294319, 0.04285084, 0.04453555, 0.04448689,\n",
       "        0.04466526, 0.04466183, 0.04467896, 0.04467945]),\n",
       " 'std_train_score': array([0.        , 0.00138524, 0.00757197, 0.00566626, 0.00992457,\n",
       "        0.00965834, 0.01033383, 0.01031565, 0.01033124, 0.01033118,\n",
       "        0.01033136, 0.01033136, 0.01033136, 0.01033136])}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the complete results (list of named tuples)\n",
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47999943493683117\n",
      "{'C': 0.1, 'penalty': 'l2'}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果最佳值在候选参数的边缘，最好再尝试更大的候选参数或更小的候选参数，直到找到拐点。 l2, c=0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZQAAAEKCAYAAAA1qaOTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VFXawPHfk0mDNAIEEggQek9AIqg0ERRElqaiu2J3UVffdW0rlnXFCvZd3VUQCyorukgQ0UWKIogiINICIkVKQiChpJI2M+f9YyYQIIFJMpOb8nz3Mzu3nHvvcxDy5Jx77zlijEEppZSqKj+rA1BKKVU3aEJRSinlFZpQlFJKeYUmFKWUUl6hCUUppZRXaEJRSinlFZpQlFJKeYUmFKWUUl6hCUUppZRX+FsdQHVq2rSpiYuLszoMpZSqVX766afDxpioc5WrVwklLi6OdevWWR2GUkrVKiKy15Ny2uWllFLKKzShKKWU8gpNKEoppbxCE4pSSimv0ISilFLKKzShKKWU8gpNKEoppbxCE4pSSimv0ISilPKafu9eSb93r7Q6DK+oK3WpznpoQlHKYnXlB5dSmlCUUkp5hSYUpZRSXqEJRSmllFdoQlFKKeUVmlCUUkp5hSUJRUQai8gSEdnh/o48S9lwEUkVkddLbVsuIttFZIP706x6IldKKVUeq1ook4FlxpiOwDL3enmeAr4tY/t1xphe7k+6L4JUSinlOasSyhhglnt5FjC2rEIi0gdoDiyuprhULXHzopu5edHNVoehlCrFqoTS3BiTBuD+PqPLSkT8gJeAB8s5x7vu7q6/iYj4LlSllFKe8Nmc8iKyFIguY9ejHp7iT8CXxpj9ZeSL64wxqSISBnwKXA+8X04ck4BJAK1bt/bw0koppSrKZwnFGDOsvH0ickhEYowxaSISA5R1D+RCYKCI/AkIBQJFJNcYM9kYk+q+Ro6I/AfoSzkJxRgzA5gBkJiYaKpWK6W87++zt7kW6kAPntal5qnOeljV5bUAuNG9fCPw2ekFjDHXGWNaG2PigAeA940xk0XEX0SaAohIADAK2FI9YSullCqPz1oo5zAV+EREbgX2AVcDiEgicIcx5razHBsEfOVOJjZgKfCWj+NVNczWtGyrQ1BKncaShGKMOQIMLWP7OuCMZGKMeQ94z72cB/TxbYRKKaUqyqoWilJVMvk/e1wLtbx/W6m6RIdeUUop5RWaUJRSSnmFdnkp5WPG6cSZm4sjKwtHZpbrOysTZ2YmjrRdtC4qwh9D2m2jAQMG3P+HKVk+5ZszyrkLnmX51GPNKceWWq7iuToVFQKQcmX/s/+h1AJ1pS4l9SjeuZGADgk+vZYmFKU8ZIqLcWRnl0oMmTiysnBmZeHIcm8/5ZOJMysbR3Y2OJ3lntff34afvyF33S8nN5Y39sMZ2+Vci+UfI+WVOfWAssehkDKPN8WuTo/CA5llHVSr1JW6lNTD5Of5/FqaUFS9YozBFBSc/KHvaWLIzMJ5/Hj5JxbBLzwcW0QEtrBQbMFCYEwwthg7fo4CbPYMbIF2bIFObKENsbXshK1NT/za9WHlE3/neL4/ly/ZWt5P71rjy2HdABi5NNniSKqurtSlpB4de17k82tpQlG1UpDJR5yGov37T+lGOpEYMrNOtiZK78vMwhQXl3/igABsjSKwhUdgi4ggIDqa4M6dXYmiUYQ7aTQ6sW4LcGAr2Idf1g7k0CY4uAmOrj55vtBoiImHmCshOh5iEqBR61MSR17+066FWp5MlNKEomodYwzhOYbGWbDr0svKLOPXsCF+jSJcP/zDwwlq1/5kEoiIwC/C9W0LP7nNFhGBNGhAmWONGgNZKa6EkbYR9m+CNRsh58DJMo3auBJGrz9ATC9XAglr7qM/BaVqHk0oqlYxdjuHnn2OJlmQFwwdnnjOnQwaYYtwdzmFhyOBgZW/iNMJR3e5EkfaRncS2QT5R137xQ+adoK4Aa4EEhMP0T2hQbnzxClVL2hCUbWGMy+P1PsfIHf5cjJD4VgEJI4tcyodzzmKIeMXd/Jwtz4ObYGiXNd+WyA06wpdR7m7rHpB8+4Q2LDqFVKqjtGEomqF4kPp7L/zDgp/2U703x/nt7efqvhJio7DoWQ4uPFkAknfCo4i1/6AEFdLo9d17vseCdC0M/hXobWjVD2iCUXVeAXbf2X/HXfgyMqi1Rv/JnTwYDhXQsnPPNlVVdJtdfhXMO7HdxtEuhJGvzvc3VYJ0Lgd+Nl8XyGl6ihNKKpGy121itR7/oJfw4bEffgBwd26nVko55A7eWw4mUAy957cH9bC1eLoNubkk1YRsTXmqaop13UFYKTFcXiD1qXmqc56aEJRNVbm3LmkPTGFoPbtaTX9TQKiT04A6m9z0rNDJrzYGXIPnjyocTto0Rv63OhKHNEJEBplQfRK1T+aUFSNY4wh4x//4Mib0wkZMICWr76CLTT0lDKtmucRE5UP7Ua7bpTHJEB0DwiOsChqpZQmFFWjOIuKSHv4EbK/+IJGV19N9ON/QwICzihn61zIbgJpN36GBVEqpcqiCUXVGPZjx0j5v/8jf91PRN1/H01uu63slwyP7aVTcSGzwyJpV/1hKqXKoQlF1QhFe/eyf9LtFKel0fLllwgfeZZbiFvns6xhA15rHMqHn15+IukIcspyCRE5sV66zNn2net4KTXAoqfHiwjGgMNpKHYYiuxOih2GPHsmGD9GfHQ9In4IfviJHyJ++JUsu79LPmesi+1EWduJ7TbklPWSZVvZy35+2Eqt28R2oozNz/0tgs3P5trn5zqnv5+rnM3PRlGRqzW5aMe6k/U/8edz8v9P/rmcXKbUEWccK6eun37Mqec59fiT+zjFqcefUhKA4uJABMPalJ34mi8fD7EXu/6b5BUVEhIY5MMraUJRNcDxn38m5U93gTG0fu9dGp53XrllCx2FvLj9A+Y0jyLICQnNElwDPpYair1k2WAw5uRyidLlT1kuNSy7KflfqePLvIb7eLvTUGR3UGR3UuRwUmy3U+RwLzucFDscFDkMdoejdCTuHyQNACf7so4AThCn6yIYECeCATGn7itZxyAntp26T6TU0PPVJdj19eD3dWAqTffP3luWjbM2jqpy12Ntyg4ubtfDp5fShKIslb1oEQf++hD+MdG0nj6dwLi4csv+lvUbD359D9tthSTmF5HrF8zUgVN9Flt+kYPDuYWk5xSSkVNIRq7r+7D7O6PU9iL7mcPTB9iEqNAgosKCXN+N3Msl6+7lq+bfgp+fnR9v/vTEscYYnAacxuA0rjlHTKl1pzvflF53JTdOrDudBofTicM4cDid2I0Dh9OO3enE4TQ4jB2704HDGJxOh3vZid3pwGlcyw6nE4fT4T6HwYkDu9Pp3u86r9M4cRjXtnc3uuowsUfJCAbmjP93LZlSU62U3s4pZdx/GGfsKz1Piyl9dnP6uc78heKU6VxK9pb6RaLE/F+XADCm0zBqs89+XQpAx6YtfH4tTSjKEsYYjr79NukvvkSD884j9l+v4x9Z9lhYxhgW7FrAMz8+Q7DT8K+D6Xwc1pQ8v4p3FBQ7nBzNKzojIZReL0kYOYX2M44XgSYhgTR1J4R2TUNOJonTEkVEg4Cy7wGdxs+vrOsINgGbTztDvG/Orx8A8NdBV1ocSdUt3v85AE8Pq92trSX7FwLQMryxz6+lCUVVO2O3c/Dpp8mc8zHhIy8n5rnn8Asqu283rziPp1c/zcLdCzk/+nye+207zRt353Vz9EQZp9OQmV98RkI4PVFk5BZyNK+ozOuEBfufSAjdWoSfSApNS7UwmoUF0TgkEH+bzpytVFk0oahq5cjNI/W+e8lbsZImf/wjUff+BfEr+wf01iNbefDbB0nJTeGuXnfxx5aXYPvhfLjsGXZv+ZGsA0O44NllHM4txO48835BkL/ficTQpklDEuMiz2hJlCSM4AAdckWpqtKEoqpN8aFD7L/9Dgp37CD6ySlETphQZjljDLO3zebln16mcXBj3r7sbRKjE2HFi64C3caQv+oYxQXNGNC9Kc3K6XIKDfL3qMtJKeUdmlBUtSj45Rf2334HztxcWr35JqEDB5RZLrMgk7+t+hvLU5ZzcauLeeqip2gU3Mi1M3k+xPbFRMRSkN2egAYHefHqUdVYC6XU2WhCUT6Xu3Kla4DH8HDa/Gc2wZ07l1lu3cF1PLTyIY4VHGNy38n8ocsfTrYwDu+AQ5th+HNsSsnCURxBw8abq7EWSqlz0buLyqeOffwJ+++4k4A2bYj7eE6ZycThdPDGxje4dfGtNPBvwIcjP+S6rted2l2VnOT67jaG+RtSQRwEh+2pnkoopTyiLRTlE8bpJOPllzky821CBg+i5UsvYwsNOaPcobxDTF45mXWH1jGq3Sgeu+AxQgLOLEdyErS+EHtoDJ9v3EZQyH78bGU/saWUsoYmFOV1zsJCDjw0mZxFi2h07TVEP/YY4n/mX7UVKSt49LtHKXQU8syAZxjdfnTZJ0z/xTWz4uXP8/2uIxzOLSSixS4f10IpVVGaUJRX2Y8dI+VPd5H/8880e/BBGt9y8xlPWhU7inll/St8sPUDOkd25oXBL9A2om35J906HxDoOpr5i1IJC/YnKCTFtxVRSlWYJQlFRBoDHwNxwB5ggjHmWBnlHEDJndd9xpjR7u1tgTlAY2A9cL0xRvs/LFa0Zw/7br8d+8FDtHz1VcJHDD+jzL7sfTy44kG2HtnK77v8nvsT7yfIdo4B65KToM1F5Ac346stG7kiPoZvcxw+qoVSqrKsuik/GVhmjOkILHOvlyXfGNPL/SndHzINeMV9/DHgVt+Gq87l+E8/seeaa3Hm5NJm1ntlJpMvd3/JhIUTSMlJ4dUhr/JIv0fOnUzSt0HGL9B9HEu3HSKvyMHYXi19VAulVFVYlVDGALPcy7OAsWcpewpx9Z9cAsytzPHK+7K++IJ9N92MLTKSuDkf0aBXr1P2Hy8+zuOrHuehlQ/RKbITc383l6Gth3p28i3zQPyg62g+25BK8/Ag+rVr4oNaKKWqyqqE0twYkwbg/m5WTrlgEVknIqtFpCRpNAEyjTElI+qlAOX+yioik9znWJeRkeGt+BWuN9oPT5/BgfsfIDghnrg5HxHYuvUpZbYf3c61X1zL/J3zmRQ/iXeGv0NMaIynF3B3d/XnmF8ky7dnMDqhBbZKDAqplPI9n91DEZGlQHQZux6twGlaG2MOiEg74GsR2Qxkl1Gu3IkfjDEzgBkAiYmJFkwQUTeZ4mLSpkwha+6nhI8aRcyzz+AXGHhyvzF8sv0Tnl/7POFB4bx12Vv0i+lXsYscSoYjO+CCO/lySxp2p2GMdncpVWP5LKEYY8qdREBEDolIjDEmTURigPRyznHA/b1bRJYDvYFPgUYi4u9upcQCB7xeAVUuR24uqX++h7zvv6fJnXcQ9ec/n/IkV1ZhFlN+mMKSvUvo37I/z/R/hiYNKtFNlZx0srvrw510aBZK9xbhXqyJUsqbrOryWgDc6F6+Efjs9AIiEikiQe7lpkB/YKtxTaH3DXDV2Y5XvlGclsbeP1xH3po1xDzzNM3uueeUZLIhfQMTPp/AN/u+4f4+9/Pvof+uXDIp6e6KG0hKcQhr9hxlbK8WOtijUjWYVQllKnCpiOwALnWvIyKJIjLTXaYrsE5ENuJKIFONMVvd+x4C7hORnbjuqbxdrdHXU/nJyeyZcA3FBw7QesZ0Gl15chIlp3Eyc/NMblp0EyLC+5e/z009bsJPKvlX7OAmOLoLeoxnwUZXA1S7u5Sq2Sx5D8UYcwQ44zEfY8w64Db38vdAz3KO3w309WWM6lQ5y5eTet/92BpF0Obt2QR36nRi3+H8wzy88mFWp61mRNwIHr/wccICw6p2weQkEBt0+R2fzUimT5tIWjVuWMVaKKV8Sd+UV+d09D//4dDTzxDcpQuxb75BQLOTD+V9n/o9D3/3MMeLj/PEhU8wvuP4qndLlXR3tRvMtuwAth/K4akx3atYC6WUr2lCUeUyTifpL7zI0XffJXTIEFq++AJ+Ia6BG4udxbz+8+u8s+UdOjTqwNuXvU2HyA7euXDaBji2Bwbez/wNqfj7CVfEt/DOuZVSPqMJRZXJmZ/Pgb8+RM6SJUROnEjzhycjNtc0uam5qfx1xV/ZlLGJqztdzYPnP0gD/wbeu3hyEvj54+w8is+/+plBnaJoHBJ47uOUUpbShKLOYD9yhP1/+hMFmzbT/JGHaXzDDSf2Ld6zmCe+fwKD4cXBLzI87swhVqrkRHfXENYcMhzIKuChy7t49xpKKZ/QhKJOUbh7N/sn3Y798GFiX/snYcNcrxMV2At4Ye0LfPLrJ8Q3jWfaoGnEhsV6P4DU9ZC5DwZP5rMNqTQMtHFpt+ZnFIsresD711ZKVYkmFHVC3po1pPzfnxF/f9q8P4sG8fEA7MrcxQPfPsDOzJ3c0uMW7u59NwF+Ab4JInke+AVQ2GEEX3y2jsu6NadhoP41Vao20H+p9czNi24G4N0R756yPWvBAg48+hiBrVvTavqbBMbGYowhaWcSz/34HA0DGvLmsDfp37K/74IzBpLnQ/tLWL6vmOwCO2N667snStUWmlDqOWMMh994g8P/fI2G/foR+89/YIuIILcolyd/eJL/7fkf/WL68dyA54hqGOXbYFLWQXYKXPIYn21IpUlIIAM7NPXtNZVSXqMJpZ659rVk18IIMEVFpP39CbKSkogYM4aYp55EAgNJPpzMA98+QFpeGvecdw+39Lil8m+8V0TyPLAFkh13GUvnruH357fC32bVYA5KqYrShFJPObKzSfnzPRxfvZqmd99N07v+hMHwfvIsXl3/KlENonhvxHv0atbr3CfzBqfT1d3VYRiLdh6nyO48a3fXx7dfWD1xKaU8pgmlHrIVO9nzhz9QtHcfMVOfo9HYsRwtOMpj3z3GytSVDG09lCkXTSEiKKL6gkpZAzkHoPsUPluTSpsmDendqlH1XV8pVWWaUOqZwAIHUQfzsYdk0HrmTEL69WVN2homr5xMVmEWj/Z7lGs6X1P9o/omJ4EtiPSYIXy/aw3/N6SDjiysVC2jCaUesR8+TLO04zj9hLiP/oOtbRte//l1ZmyaQZvwNrwx7A06N+5c/YGVdHd1vJQFv+RgDPXq6a4fb/7U6hCU8gpNKPVIxmuvI05Ib9mQqOgQHvrqVtanr2dsh7E83PdhGgZYNJrvvh8g9yB0H8dnyw/Qs2UE7aNCrYlFKVVpmlDqiYJffyXzv/8lJyKAjXGGv39+FcWOYp4b+Byj2o2yNrjkJPAPZnfjgWxOXcdjV3S1Nh5VaXWptVVX6lKd9dCEUk+kP/8CfmGhfDKgkMXxRXQNac+Lg1+kdXhrawNzOmDrZ9DxMuYnZ+InMDpBRxZWqjbSh/zrgdyVK8n77juyfz+cxfEOLthl48ORH1qfTAD2fg956Zju45i/4QAXtW9Ks/Bgq6NSSlWCJpQ6ztjtpD//PAGtW/N0y58ItgVTdGECgbYaMhx8chIENGRjg37sO3qcMb20daJUbeVRQhGR/iIS4l6eKCIvi0gb34amvCFz7qcU7thJ8oTz2H18H63CWlXPW++ecNhd3V2dhpO05RhB/n6M6BFtdVRKqUry9CfLG8BxEUkA/grsBd73WVTKKxy5uWS89hoBvRN4rsHXDIodVL0vK57L3u/g+GHsXceycFMaw7o2JyzYR6MYK6V8ztOEYjfGGGAM8A9jzD+AMN+FpbzhyIy3cBw5wsJRTSl0FvFg4oNWh3Sq5CQICGGV9OZIXpF2dylVy3n6lFeOiDwMTAQGiYgN0F8la7Di1FSOvvceZvgg3rGv4MbuNxIXEWd1WCc57LB1AXQeQdLmo4QH+zO4s49HM1ZK+ZSnLZRrgELgVmPMQaAl8ILPolJVlv7KqyDC630ziQyO5Pb4260O6VR7VkD+UQo7j2bx1kNcER9DkL/N6qiUUlXgaULJwdXVtVJEOgG9gI98F5aqivxNm8heuJCj4way0r6Vv5z3F0IDa9ib58lJEBjK4qJ4jhc5GNOr/gy1olRd5WlCWQEEiUhLYBlwM/Cer4JSlWeM4dDUafg1acLTHbbQrUk3xnQYY3VYp3IUw7bPofNI5m06TIuIYPrGNbY6KqVUFXmaUMQYcxwYD7xmjBkHdPddWKqycr5aTP769WwZ35P9jsM83PfhmvOYcInd30L+MbLbj2LFjsP8rlcL/Px0ZGGlajtPb8qLiFwIXAfc6t6mHd41jLOoiPQXX8SvQ1uea/ojI9uOPGOCrNPnkrdEchIEhbMgtwsO507GaneXUnWCp7+6/gV4GEgyxiSLSDvgG9+FpSrj2IezKU5JYeHIpvj5+3Nvn3utDulM9iL45WR3V+fmYXSNCbc6KqWUF3iUUIwx3xpjRgP/FpFQY8xuY8yffRybqgD7sWMcfuMN7P0SeC/kZ27tcSvRITXwrfPdy6Egi/TWV7B+XyZjeuu7J0rVFZ4OvdJTRH4GtgBbReQnEan0PRQRaSwiS0Rkh/s7spxyDhHZ4P4sKLX9PRH5rdS+apr4vOY6/Pq/cB4/zr8G5NIytCU3dr/R6pDKljwPgiL477EOgI4srFRd4mmX13TgPmNMG2NMa+B+4K0qXHcysMwY0xHXU2OTyymXb4zp5f6MPm3fg6X2bahCLLVe4e7dHJszh8OX9mZV4F7uT7yfYP8aOGKvvRB++QLT9QrmbUqnb1xjYiMtmtRLKeV1niaUEGPMiXsmxpjlQEgVrjsGmOVengWMrcK56r30F15EGgTzXI/dnB99PsNaD7M6pLLt+hoKs9kbPZxdGXna3aVUHeNpQtktIn8TkTj35zHgtypct7kxJg3A/d2snHLBIrJORFaLyOlJ5xkR2SQir4hIUBViqdXyVq8m95tv2DKyM6mBuTx0/kOI1NBHcJOTILgRH2W0JcAmXNEzxuqIlFJe5Oljw7cAU4B5gOB60fHmsx0gIkuBsu4KP1qB+FobYw64nyr7WkQ2G2N24Xri7CAQCMwAHgKeLCeOScAkgNata8CEUl5kHA4OTXseopsxLW4LV3e6ms6NO1sdVtmKC+CXL3F2H8v8zRkM7tSMRg1ryJwsSimv8CihGGOOARV6qssYU26/i4gcEpEYY0yaiMQA6eWc44D7e7eILAd6A7tKWjdAoYi8Czxwljhm4Eo6JCYmmorUoabLmv8Zhdu28cWNnQloUMxdve6yOqTy7VwKRTlsixzKoexC/jZKu7uUqmvOmlBE5HOg3B/CZdwo99QC4EZgqvv7szKuHQkcN8YUikhToD/wvHtfSTISXPdftlQyjlrLefw4Ga++SlHXtsyK2cnkXg8TGVzmw3I1Q3ISNGjMB4daExp0mGFdm1sdkVLKy87VQnnRR9edCnwiIrcC+4CrAUQkEbjDGHMb0BWYLiJOXPd6phpjtrqPny0iUbi63zYAd/gozhrryNvvYM/I4I3xNto36sCEzhOsDql8xfmw/X/Yu1/JFz8fZnj3aIIDdKAFpeqasyYUY8y3vrioMeYIMLSM7euA29zL3wM9yzn+El/EVVsUHzrEkbff5vCFnVjVeDcz+j5LgF8Nnp5mxxIozmN96MXkFNoZq093KVUneXQPRUQ2c2bXVxawDnjanSBUNcl49R8Yh4Pn+qQypNUQLmxxodUhnV1yEjRsyrsHYokKy+Gi9k2tjkgp5QOePuX1P8AB/Me9fi2u7qYsXMPY/87rkaky5ScnkzV/PsnDO3IwYj/Ta9q0vqcryoNfF1HYfQLL1h5l4gVtsOnIwkrVSZ4mlP7GmP6l1jeLyCpjTH8RmeiLwNSZjDGkT3seEx7K8113cUO322gV3srqsM5ux2IoPs6qoIEUOZw6b7xSdZinLzaGiki/khUR6QuUTAFo93pUqky533zD8TVrWDS0EaGRzfhj/B+tDunckpMgJIq39kXTtmkI8bERVkeklPIRT1sotwHviEgorq6ubOBWEQkBnvNVcOokU1xM+vMvUBQbxayOB3iqz7OEBFRl9JtqUJgLvy4mr/s1rF6TxT1DO9bct/iVUlXm6YuNa4GeIhKBa/bGzFK7P/FJZOoUx+Z8TNGePcy8LpLuzeIZ1W6U1SGd246vwJ7PMr/+GINOpKVUHefpU14RwN+BQe71b4EnjTFZPoxNuTmysjj8+usc6dGS5a0OMrvv5Jo3rW9ZtsyD0Gim/9aMhFYBxDWt4S0qpVSVePpT6R0gB5jg/mQDNWAu2frh8Btv4sjO5sULDjO6wxjio+KtDuncCnNgxxKOtb2c5IN5jNWb8UrVeZ7eQ2lvjLmy1PoUEanXc5BUl6J9+zg6ezbbLmzBgZhsZpx3j9UheWb7InAU8qXjAmx+wqh4TShK1XWetlDyRWRAyYqI9AfyfROSKi39xZcwNj9e6XOQSfGTaNawvJH+a5jkJExYC97Y1ZT+HZoSFVZvZxhQqt7wtIVyJzCr5KY8cBS4yVdBKZfj69aRs3gxSy9tQmh0GNd3u97qkDxTkA07l3Co80RS1hdy33BtnShVH3j6lNcGIEFEwt3r2T6NSmGcTg5Ne56iJmHMSsjk+fOnEGSrJb/lb/8SHEXML+pLcIAfl3Uva1ocpVRdc67h6+8rZzsAxpiXfRCTArK/+IKCzZv5YGwI57W+kEta1aLxMJOTMOEtmb4rkku7NSc0yNOGsFKqNjvXv/SwaolCncJZUED6y69wrE1jlnXN5b81eVrf0+Vnws5l7OswkWPpDn26S6l65FzD10+prkDUSUffm4U9LY1/XufPhC5/oENkB6tD8tz2L8FZzH8L+hLZMIBBnaKsjkgpVU0q/HaciKz3RSDKxZ6RwZEZM9jRszEpHSP4U68/WR1SxSQn4YxoxczfGnFFfAwBtlrwAqZSyisq86+9lvS91E4Zr72Oo7CA1y7K4u5edxMRVIsGU8w/Bru+ZmfTYRQUGx1qRal6pjJ3S7/wehQKgILtv5I5dy4rLgghrH0sV3W6yuqQKmbbQnDamZ3bh9jIBvRpU4PnuFdKeV2FWyjGmMd8EYiC9Oefp7hBAO/1Pc7k8yfj71fLno5KTsIR0YYP9kV8yN4LAAAe0UlEQVQypleL2vMggVLKKzxKKCKSIyLZp332i0iSiLTzdZD1Qe7KleStWsUn/eGiLpfRN6av1SFVzPGjsHs5yZFDcRrR7i6l6iFPfwV+GTiAawpgwTUFcDSwHdfAkRf7Irj6wtjtHJo2jexmISw+z8m8PmW+/lOzbfscjIN3MnvTLSacjs31iXOl6htPu7xGGGOmG2NyjDHZxpgZwEhjzMeAdpRXUebcuRTt3MX0AflMjL+J2LBYq0OquOQkiiPaMv9gY53mV6l6ytOE4hSRCSLi5/5MKLXP+CKw+sKRm0vGP19jT7sQ9vZqzm09b7M6pIrLOwy/reDnsIsREUZrQlGqXvI0oVwHXA+kA4fcyxNFpAFwt49iqxeOTJ+B4+hR3hxUwL2J99EwoKHVIVXctgVgHMw4kkC/to2JiWhgdURKKQt4OjjkbuB35ez+znvh1C/FqakcmTWL1QnBhMf34Iq2V1gdUuUkJ1EQ0Z6lh6KYOkRvxitVX3n6lFcnEVkmIlvc6/Eioo8PV1H6y6/gwMGsAcVM7ju5dj5mm5sOe77jx4aDCLTZuLxnjNURKaUs4mmX11vAw0AxgDFmE64nvVQl5W/cSPYXX7DgfBh43ji6N+1udUiVs20BGCf/To9nSJcoIhoEWB2RUsoiniaUhsaYNadts3s7mPrCGMOhqdPICw/kq4Eh3FNbpvUty5Yk8sLb82NeM333RKl6ztOEclhE2uN+oktErgLSfBZVHZfz1Vfk//wzH/S3c1PiHTRt0NTqkCon5yDsXcV3gQMJCw5gSJdaMj2xUsonPH2x8S5gBtBFRFKB33A9+aUqyFlUxKEXXyQtOpDdA2J5setEq0OqvK0LAMO/0ntyeXw0wQE2qyNSSlnI0xZKKvAu8AwwB1gC3FjZi4pIYxFZIiI73N9lvhwpIq1FZLGIbBORrSIS597eVkR+dB//sYgEVjaW6nbsgw+xp6Qy82I7D/T9KwG2WnzPITmJ7PCObCqK0e4upZTHCeUzXI8NF+MagiUXyKvCdScDy4wxHYFl7vWyvA+8YIzpCvTF9R4MwDTgFffxx4BbqxBLtbEfO0bGG2+wqYM/4f0HMih2kNUhVV72Adj3A1/79ad5eBD92jWxOiKllMU87fKKNcaM8OJ1x3By/K9ZwHLgodIFRKQb4G+MWQJgjMl1bxfgEuAPpY5/AnjDi/H5xOHX/4XjeB4fDAngn+f/tXY+Jlxi62eA4Y2MHoy+qAU2v1pcF6WUV3jaQvleRHp68brNjTFpAO7vsu7mdgIyRWSeiPwsIi+IiA1oAmQaY0qeMksByu1vEZFJIrJORNZlZGR4sQoVU7h7N8fmfMSSXsLAgX+gXaNaPkhzchLHwjqx3dGCMdrdpZTC8xbKAOAmEfkNKMQ14rAxxsSXd4CILMU1IvHpHq1AbAOB3sA+4GPgJmBBGWXLHU/MPZDlDIDExETLxh079PwLFAQIi4c2Yk6vO60KwzuyUmD/jywKvYkOzULp3iLc6oiUUjWApwnl8oqe2BgzrLx9InJIRGKMMWkiEsPJeyOlpQA/u4d9QUTmAxfgGi6/kYj4u1spsbju69RYeT/8QN7y5cwd4sfNA+8hPLCW/wDe+hkA04/05KpLdSItpZSLR11expi9ZX2qcN0FnHxK7EZcN/1PtxaIFJEo9/olwFZjjAG+Aa46x/E1gnE4SJs6lSONbOy+tAvjO4y3OqSqS04iI7Qze0yMdncppU6o8BTAXjIVuFREdgCXutcRkUQRmQlgjHEADwDLRGQzrm62t9zHPwTcJyI7cd1Tebua4/dY1vz5FG//lfcHG+6/6GFsfrX8XY3MfZCylgXF/ejTJpJWjWvh6MhKKZ+wZNJyY8wRYGgZ29cBt5VaXwKccZ/G3Q1W4+fIdeblcfCVV9jZ0o/wy0eQGJ1odUhVlzwfgPeyezPpYp33RCl1klUtlHrhyNvvYA4fYfalgdyf+IDV4XhHchJpIV1Jk2iuiNeEopQ6SROKjxQfPEjG2zNZ1VUYNOKPxITWgWHdj+2BA+v5tOB8BnZsSuOQWjNAgVKqGmhC8ZH0V1/FYS9myeXNubnHzVaH4x3u7q45x/swtrfejFdKnUoTig/kJyeTPf8zvjgfbr70IRr415EpcZPnsb9BV44GRHNpt+ZWR6OUqmE0oXiZMYYDU58lp6Gwe3RvhscNtzok7ziyC9I28nF+Ipd1a07DQEue51BK1WCaULws9+uvKVq7nk8GCPcOfqzuvPS31dXdNa/gfMZod5dSqgz6a6YXmaIiUqc+S0oTIezqK+napKvVIXlPchK7g7tRaGvBwA61dEIwpZRPaQvFi47OmYPZf4C5l4Vwd2Itntb3dId3wsHNfJSXyKj4GPxt+tdGKXUmbaF4iSMzk4Ov/YMtcUL/K++mSYM6ND9IchIAnxefz7+1u0vVUsXFxaSkpFBQUGB1KDVWcHAwsbGxBARUbuI/TShecujf/4bc4yy9uRX/7vqHcx9QmyQnsT2wO0EhrejdqpHV0ShVKSkpKYSFhREXF1d37m16kTGGI0eOkJKSQtu2bSt1Du278IKivXs59p/ZfBMv3Djm8do9re/pMrZDejIf5SUyJkFHFla1V0FBAU2aNNG/w+UQEZo0aVKlFpwmFC/YP+05isTJ3msuYkDLAVaH413J8zEIXzr66tNdqtaraDK5ZvoPXDP9Bx9FU/NUNdlqQqmi4+vWUfT1tyy4yJ+7hj5mdTjel5xEsn93mreMo31UqNXRKFWrhYae/Dc0YsQIGjVqxKhRo8ose9ddd9GrVy+6detGgwYN6NWrF7169WLu3LkVuub69etZtGhRleL2lN5DqQLjdLLn6Sc4GgbhN0wkLiLO6pC8K30bZGzj4+KbGDNYB4JUypsefPBBjh8/zvTp08vc/69//QuAPXv2MGrUKDZs2FCp66xfv54tW7YwYsSISsfqKW2hVEHW558jv+xiwaVh3Hb+n6wOx/uSk3Dix1fOvoxO0ISilDcNHTqUsLCwSh27Y8cOhg8fTp8+fRg0aBC//vorAHPmzKFHjx4kJCQwZMgQ8vPzefLJJ5k9e3alWjcVpS2USnLm57P/hefYGw0X3vgQYYGV+4tRYxmDSU5ig193OrXvQLPwYKsjUsprpnyezNYD2ecstzXNVcaT+yjdWoTz9991r3Jsnpg0aRIzZ86kffv2rFq1irvvvpvFixczZcoUli9fTvPmzcnMzKRBgwY8/vjjbNmyhVdffdXncWlCqaRD78zE/3AWK+5sx9RO46wOx/vStyKHf+XT4lsY00tbJ0rVFJmZmaxevZorr7zyxDa73Q5A//79ueGGG7j66qsZP776pxvXhFIJ9owMDs+YwU+dhN///mn8pA72HLq7u76WfkzuEW11NEp5lactiZKWyce3X+jLcCrEGEPTpk3LvKfy1ltv8eOPP7Jw4UISEhLYtGlTtcZWB38S+t6el6dCsZ2U64fQu1lvq8PxPmMwW5JYS3fO69qJsOA69F6NUrVcZGQkMTExJCW5RrBwOp1s3LgRgN27d3PBBRfw1FNPERkZSWpqKmFhYeTk5FRLbJpQKqhg+3YK53/JsvMDuG3k36wOxzcObkaO7mR+cV/t7lLKRwYOHMjVV1/NsmXLiI2N5auvvvL42Dlz5vDmm2+SkJBA9+7dWbhwIQD33nsvPXv2pGfPngwbNowePXpwySWXsHHjRnr37q035WsSYww7nnqUgiAIm3Qr0SF1tCsoOQkHfqwKuJAnOkdZHY1SdUZubu6J5ZUrV3p0TFxcHFu2bDllW7t27cpMQAsWLDhjW1RUFOvWratgpJWjCaUCsr9djv+6ZJaMbMR9/W63OhzfMAZnchI/mB70T+hMkL/N6oiUskxNundSG2iXl4eM3c7upx8nLRIS7/wbwf519DHatI34HfuNBfZ+jOmlQ60opTynCcVDaR99QHDKYVaP78ylHS63OhzfSU7Cjo1NIQPoG9fY6miUUrWIdnl5wJGTQ/o//8muVsL4W6bW3dFKjcGxeR6rnD0Y3Lszfn51tJ5KKZ/QFooHll/Rn6CcAtJuHU6XJl2sDsd3DqzHlr2Pzx0XMFa7u5RSFaQtFA/sauZgRzPhurGPWx2KbyUnUYw/vzW5mK4x4VZHo5T13r3C9X3zF9bGUUtoC8UDS27oxv+u60BkcKTVofiOMdg3z2OFoydDz+tkdTRK1UnVPXx9UlISL7zwQpXj9pS2UDwQGhgK1PG5QFJ/wj8nlS8cv+M+HVlYKZ/z1vD1drsdf/+yf5SPG1e94wxaklBEpDHwMRAH7AEmGGOOlVGuNTATaAUYYKQxZo+IvAcMBrLcRW8yxlRusgAPvDviXV+dusYwWz6lGH+OxF5KbGRDq8NRqs4bOnQoy5cvr9SxAwYMYPDgwaxcuZLx48fTtm1bnn32WYqKioiKiuLDDz+kWbNmzJw588RIwxMnTqRJkyasXbuWgwcP8tJLL3k94VjVQpkMLDPGTBWRye71h8oo9z7wjDFmiYiEAs5S+x40xvh2HIH6wunEvjmJbx0JXNano9XRKOV7/5sMBzefu9xB9+CKJfdSzia6J1w+tWpxVUB2djYrVqwA4NixY4wePRoR4c033+Sll15i2rRpZxyTnp7OqlWr2Lx5MxMmTKgzCWUMcLF7eRawnNMSioh0A/yNMUsAjDG5KN9IWUtAXhqLzHj+1jPG6miUUh649tprTyzv27ePCRMmcPDgQQoLC+nUqez7oGPHjkVEiI+PJzU11esxWZVQmhtj0gCMMWki0qyMMp2ATBGZB7QFlgKTjTEO9/5nRORxYJl7e2F1BF4XOZPnUUwAhe2H06hhoNXhKOV7nrYkavBTXiEhISeW77rrLh555BFGjhzJ0qVLmTq17PoFBQWdWDbGeD0mnz3lJSJLRWRLGZ8xHp7CHxgIPACcD7QDbnLvexjo4t7emLK7y0rimCQi60RkXUZGRmWrU3c5nRRvSmK5I4ER2t2lVK2UlZVFy5YtMcYwa9Ysy+LwWUIxxgwzxvQo4/MZcEhEYgDc3+llnCIF+NkYs9sYYwfmA+e5z51mXAqBd4G+Z4ljhjEm0RiTGBWlI+eeYf9qgvIPsdTvIoZ1bW51NErVG1UZvv50TzzxBOPGjWPw4ME0b27dv2OrurwWADcCU93fn5VRZi0QKSJRxpgM4BJgHbiSkLurTICxwJYyjlcesG+eh90E4N91JMEBOrKwUr7kreHrv/vuu1PWr7zyylOmBC5x2223nVj+8MMPy43FW6xKKFOBT0TkVmAfcDWAiCQCdxhjbjPGOETkAWCZO3H8BLzlPn62iEQBAmwA7qj2GtQFTgf2LfP52tmbkX06WB2NUjVPDbx3UpNZklCMMUeAoWVsXwfcVmp9CRBfRrlLfBpgfbHvB4ILMlgRcD3PtG9qdTRKqVpO35Svxwo3forDBBEefwU2HVlYKVVFOpZXfeWwY5Jd3V1XaHeXUsoLNKHUV3tXEVx0lLUhg4mPjbA6GqVUHaBdXvVU3s9zwQQR1XtU3Z0wTKkqunnRzUD9GM/PG7SFUh857Pj9soBlzvMY1ae91dEoVW+UDF+/YcMGLrzwQrp37058fDwff/zxGWW9MXw9wPr161m0aJFX4j8XbaHUN+9eAfmZNCjOJDlyKKObhpz7GKWUVzVs2JD333+fjh07cuDAAfr06cPw4cNp1KjRiTKeDl9/LuvXr2fLli2MGDHCK7GfjbZQ6qHMnGxyTTAtE39ndShK1UudOnWiY0fXUEctWrSgWbNmVGRoqB07djB8+HD69OnDoEGD+PXXXwGYM2cOPXr0ICEhgSFDhpCfn8+TTz7J7NmzK9W6qShtodQ3xklQfjpfOftwee+2VkejlCWmrZnGL0d/OWe5kjIl91LOpkvjLjzUt9xhBcu1Zs0aioqKaN/e8+7nSZMmMXPmTNq3b8+qVau4++67Wbx4MVOmTGH58uU0b96czMxMGjRowOOPP35iThRf04RSz5j8LBpQwG/NL2NsWNC5D1BK+UxaWhrXX389s2bNws/Psw6jzMxMVq9efcpQK3a7HYD+/ftzww03cPXVVzN+/HifxHw2mlDqmT0ZmTQxDWjbT7u7VP3laUvCl095ZWdnc8UVV/D0009zwQUXeHycMYamTZuWeU/lrbfe4scff2ThwoUkJCSwadMmb4Z8TnoPpT4oLoBtC2HuLcSag3zt7M2w+DZWR6VUvVVUVMS4ceNOtCYqIjIykpiYGJKSkgBwOp1s3LgRgN27d3PBBRfw1FNPERkZSWpqKmFhYeTk5Hi9DmXRhFJX2Ytg+yKYNwnzQnv4+Dpyty1lrmMQy8z5hAZp41Qpq3zyySesWLGC995778TjwBV5imvOnDm8+eabJCQk0L17dxYuXAjAvffeS8+ePenZsyfDhg2jR48eXHLJJWzcuJHevXvrTXlVAY5i+O1b2JKE2fY5UphFnl8Y/7Mn8pn9ArYG9aK3bOD6oBVWR6pUvVQyZPzEiROZOHGiR8eUNXx9u3btypw/ZcGCBWdsi4qKYt26dZWItuI0odR2Djvs/Q62zMO59XP8Co5yXBqyyN6HBY4L2BmSyNCEWO7sHk3fto3ZPu0ZqyNWqtbQN+QrRhNKbeR0wL4fIDkJx5b52PIPky/BLLafx0LHBeyNvJBLerTmLz2iiW8ZgV+pkYS7x+i4XUop39CEUls4nZCyFrPlU+xbkgg4nk4BgSx19GahYyLpzQdwcY84HuwRTcdmoTo+l1Kq2mlCqcmMgQPrMZvnUbx5HoF5BygigOWOBBY6J5DZ8hIG92zLo92jadW4odXRKqXqOU0oNY0xcHATjs2fUrxpHsG5+7HjzwpHT/5nxpITdxkX92zP37o1o1lYcMXPr1OaKqV8RBNKTWAMpG/FvulTCjd+SkjuHpzYWO3owWK5goJ2lzMooSOPd2lGRIMAq6NVqt7Ye/0NALT54H2LI6kdNKFYKeNXCjb+l+KNcwnL2Y0Y4WdnN5bZJlHccRQDe3Xmbx2jaBBoszpSpZQXhIaGkpuby4YNG7jzzjvJzs7GZrPx6KOPcs0115xS9q677mLVqlUUFRXx22+/0blzZwAee+wxrrrqKo+ul5SUxM6dO3nwwQe9XpeyaEKpbkd2kffzfyna+CmROb8SaIQNpgvf+v8RZ9ffMaBXNx5p14QAm75zqlRd5c3h6+12O/7+Zf8oHzdunPeDPwtNKNXh2F4y132CfdOnNM3ZRgiwzdmJVUG3Id3H0L93Dx5sFXnK471KqbqrU6dOJ5ZLD19fOqGczYABAxg8eDArV65k/PjxtG3blmeffZaioiKioqL48MMPadasGTNnzjwx0vDEiRNp0qQJa9eu5eDBg7z00kteTziaUHzEZO7n8JpPcGz+lOicZBoBG5ztmdfwFvx6jKN/n178OTpMH+9VygIHn32Wwm3nHr6+4BdXmZJ7KWcT1LUL0Y88UuFYKjN8PbgGl1yxwjXqxbFjxxg9ejQiwptvvslLL73EtGnTzjgmPT2dVatWsXnzZiZMmKAJpSYz2Wkc+H4OJnkesTmbiAK2OONYEnYztp7juSixD5N0hkSllFtlhq8vce21155Y3rdvHxMmTODgwYMUFhae0gIqbezYsYgI8fHxpKamVin2smhCqSJ79iH2fTcHv61JtM7dQEsMvzhb8d9GNxIQfyUX9u3H9eGVeLxXKeUznrYkfPmUV2WHry8REnLyl9O77rqLRx55hJEjR7J06VKmTp1a5jFBQSfnQDLGVDzoc9CEUgkFWRn89t0cbFvn0z7vJ9ph2GVasDDyeoJ7XUnfvhfRpWGg1WEqpWqoqgxfX5asrCxatmyJMYZZs2Z5IcLK0YTiodysI+xcMYeAbfPplPcTXcXBXhPN0qbXEdL7as47vz/tg/QdEaXUuZUMX3/kyBHee+89gBND2VfGE088wbhx44iNjaVv376kpaV5MVrPiS+aPTVVYmKiqcwwzj++dgO9D39BoNhJpRk7o4YRct4E4hMHERig74goVRts27aNrl27VuiY+vhiY1l/TiLykzEm8VzHagvFA87wVqy3XUVY4gS69BlCS31HRKl6oT4lEm/QhOKBC2/UOUSUUupcLPlVW0Qai8gSEdnh/o4so8wQEdlQ6lMgImPd+9qKyI/u4z8WEb0DrpRSFrOq72YysMwY0xFY5l4/hTHmG2NML2NML+AS4Diw2L17GvCK+/hjwK3VE7ZSqjarT/eMK6Oqfz5WJZQxQMmzbbOAsecofxXwP2PMcXG9Wn4JMLcCxyul6rng4GCOHDmiSaUcxhiOHDlCcHDl35uz6h5Kc2NMGoAxJk1Emp2j/LXAy+7lJkCmMcbuXk8BWpZ3oIhMAiYBtG7dukpBK6Vqr9jYWFJSUsjIyLA6lBorODiY2NjYSh/vs4QiIkuB6DJ2PVrB88QAPYGvSjaVUazcXzmMMTOAGeB6bLgi11ZK1R0BAQG0bdvW6jDqNJ8lFGPMsPL2icghEYlxt05igPSznGoCkGSMKXavHwYaiYi/u5USCxzwWuBKKaUqxap7KAuAG93LNwKfnaXs74GPSlaMqwP0G1z3VTw5XimlVDWwKqFMBS4VkR3Ape51RCRRRGaWFBKROKAV8O1pxz8E3CciO3HdU3m7GmJWSil1FvVq6BURyQD2VvLwpri62+qCulKXulIP0LrUVHWlLlWtRxtjTNS5CtWrhFIVIrLOk7FsaoO6Upe6Ug/QutRUdaUu1VUPHZRKKaWUV2hCUUop5RWaUDw3w+oAvKiu1KWu1AO0LjVVXalLtdRD76EopZTyCm2hKKWU8gpNKBUgIk+JyCb3cPqLRaSF1TFVloi8ICK/uOuTJCKNrI6pMkTkahFJFhGniNTKp3FEZISIbBeRnSJyxsjbtYWIvCMi6SKyxepYqkJEWonINyKyzf136x6rY6osEQkWkTUistFdlyk+vZ52eXlORMKNMdnu5T8D3Ywxd1gcVqWIyGXA18YYu4hMAzDGPGRxWBUmIl0BJzAdeMAYU/E5ni0kIjbgV1wv+KYAa4HfG2O2WhpYJYjIICAXeN8Y08PqeCrLPRxUjDFmvYiEAT8BY2vpfxMBQowxuSISAHwH3GOMWe2L62kLpQJKkolbCGcZlLKmM8YsLjVi82pcY6LVOsaYbcaY7VbHUQV9gZ3GmN3GmCJgDq7pHWodY8wK4KjVcVSVMSbNGLPevZwDbOMsI5rXZMYl170a4P747OeWJpQKEpFnRGQ/cB3wuNXxeMktwP+sDqKeagnsL7V+1ukYVPVyD//UG/jR2kgqT0RsIrIB1yC8S4wxPquLJpTTiMhSEdlSxmcMgDHmUWNMK2A2cLe10Z7dueriLvMoYMdVnxrJk3rUYhWajkFVHxEJBT4F/nJa70StYoxxuGe+jQX6iojPuiOtmmCrxjrbsPun+Q/wBfB3H4ZTJeeqi4jcCIwChpoafDOtAv9NaqMUXAOgltDpGGoA9/2GT4HZxph5VsfjDcaYTBFZDowAfPLghLZQKkBEOpZaHQ38YlUsVSUiI3CN2jzaGHPc6njqsbVARxFpKyKBuGYnXWBxTPWa+0b228A2Y8zL5ypfk4lIVMkTnCLSABiGD39u6VNeFSAinwKdcT1VtBe4wxiTam1UleMe+j8IOOLetLo2PrEmIuOA14AoIBPYYIwZbm1UFSMiI4FXARvwjjHmGYtDqhQR+Qi4GNfItoeAvxtjat3UEiIyAFgJbMb1bx3gEWPMl9ZFVTkiEg/MwvV3yw/4xBjzpM+upwlFKaWUN2iXl1JKKa/QhKKUUsorNKEopZTyCk0oSimlvEITilJKKa/QhKKUF4lI7rlLnfX4uSLSzr0cKiLTRWSXe6TYFSLST0QC3cv6YrKqUTShKFVDiEh3wGaM2e3eNBPXYIsdjTHdgZuApu5BJJcB11gSqFLl0ISilA+IywvuMcc2i8g17u1+IvJvd4tjoYh8KSJXuQ+7DvjMXa490A94zBjjBHCPSPyFu+x8d3mlagxtMivlG+OBXkACrjfH14rICqA/EAf0BJrhGhr9Hfcx/YGP3Mvdcb317yjn/FuA830SuVKVpC0UpXxjAPCRe6TXQ8C3uBLAAOC/xhinMeYg8E2pY2KADE9O7k40Re4JoJSqETShKOUbZQ1Lf7btAPlAsHs5GUgQkbP9Gw0CCioRm1I+oQlFKd9YAVzjntwoChgErME1BeuV7nspzXENplhiG9ABwBizC1gHTHGPfouIdCyZA0ZEmgAZxpji6qqQUueiCUUp30gCNgEbga+Bv7q7uD7FNQfKFmA6rpkAs9zHfMGpCeY2IBrYKSKbgbc4OVfKEKDWjX6r6jYdbVipaiYiocaYXHcrYw3Q3xhz0D1fxTfu9fJuxpecYx7wsDFmezWErJRH9CkvparfQvekR4HAU+6WC8aYfBH5O6455feVd7B7Iq75mkxUTaMtFKWUUl6h91CUUkp5hSYUpZRSXqEJRSmllFdoQlFKKeUVmlCUUkp5hSYUpZRSXvH/RDXShjXXn3wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#pd.DataFrame(grid.cv_results_).to_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#cvresult = pd.DataFrame.from_csv('LogisticGridSearchCV_Otto.csv')\n",
    "#test_means = cv_results['mean_test_score']\n",
    "#test_stds = cv_results['std_test_score'] \n",
    "#train_means = cvresult['mean_train_score']\n",
    "#train_stds = cvresult['std_train_score'] \n",
    "\n",
    "\n",
    "# plot CV误差曲线\n",
    "test_means = grid.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "\n",
    "# plot results\n",
    "n_Cs = len(Cs)\n",
    "number_penaltys = len(penaltys)\n",
    "test_scores = np.array(test_means).reshape(n_Cs,number_penaltys)\n",
    "train_scores = np.array(train_means).reshape(n_Cs,number_penaltys)\n",
    "test_stds = np.array(test_stds).reshape(n_Cs,number_penaltys)\n",
    "train_stds = np.array(train_stds).reshape(n_Cs,number_penaltys)\n",
    "\n",
    "x_axis = np.log10(Cs)\n",
    "for i, value in enumerate(penaltys):\n",
    "    #pyplot.plot(log(Cs), test_scores[i], label= 'penalty:'   + str(value))\n",
    "    pyplot.errorbar(x_axis, test_scores[:,i], yerr=test_stds[:,i] ,label = penaltys[i] +' Test')\n",
    "    pyplot.errorbar(x_axis, train_scores[:,i], yerr=train_stds[:,i] ,label = penaltys[i] +' Train')\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'neg-logloss' )\n",
    "pyplot.savefig('LogisticGridSearchCV_C.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上图给出了L1正则和L2正则下、不同正则参数C对应的模型在训练集上测试集上的正确率（score）。可以看出在训练集上C越大（正则越少）的模型性能越好；但在测试集上当C=0.1时性能最好（L1正则和L2正则均是）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the correct classification fraction of the zero_one_loss of LogisticRegression on test is 0.7337662337662337\n",
      "the correct classification fraction of the zero_one_loss of LogisticRegression on train is 0.7850162866449512\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import zero_one_loss  #评价逻辑回归预测模型的性能\n",
    "# 预测\n",
    "y_test_pred_lr = grid.predict(X_test)\n",
    "y_train_pred_lr = grid.predict(X_train)\n",
    "# 使用zero_one_loss评价模型在测试集和训练集上的性能，并输出评估结果\n",
    "#测试集zero_one_loss(y_true, y_pred, normalize=False)\n",
    "print ('the correct classification fraction of the zero_one_loss of LogisticRegression on test is', 1 - zero_one_loss(y_test, y_test_pred_lr))\n",
    "#训练集\n",
    "print ('the correct classification fraction of the zero_one_loss of LogisticRegression on train is', 1 - zero_one_loss(y_train, y_train_pred_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV发现的最佳参数做预测，测试集上正确分类的比例是0.7337662337662337，训练集上正确分类的比例是0.7850162866449512"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用LogisticRegressionCV实现正则化的 Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L1正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[1, 10, 100, 1000], class_weight=None, cv=5,\n",
       "           dual=False, fit_intercept=True, intercept_scaling=1.0,\n",
       "           max_iter=100, multi_class='ovr', n_jobs=1, penalty='l1',\n",
       "           random_state=None, refit=True, scoring='neg_log_loss',\n",
       "           solver='liblinear', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [1, 10,100,1000]\n",
    "\n",
    "# LogisticRegressionCV比GridSearchCV快 正则项通过参数设置，一次只能调试L1或者L2\n",
    "lrcv_L1 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l1', solver='liblinear', multi_class='ovr')\n",
    "lrcv_L1.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.46346118, -0.461086  , -0.46086753, -0.46084532],\n",
       "        [-0.46425893, -0.46462801, -0.4646866 , -0.46468586],\n",
       "        [-0.56469787, -0.56847263, -0.56880211, -0.56883287],\n",
       "        [-0.4436576 , -0.44397698, -0.44402763, -0.44403439],\n",
       "        [-0.46582886, -0.46606874, -0.46612431, -0.46612986]])}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8VfW19/HPSiAQIASBMCXBgIDMiEbE2qqVanEAn9tiS+1Eb3u1fa61g22vPNfqFeu1tVY72Vbbam1vr7a1EwJqnVBrHYhWGRLAMIdBJkkIU0iynj/2TowxyTlgdvY5yff9euWVs/f+7d9ZPw7Z6+xxmbsjIiLS3jLiDkBERDonJRgREYmEEoyIiERCCUZERCKhBCMiIpFQghERkUgowYiISCSUYEREJBJKMCIiEolucQcQp4EDB3pRUVHcYYiIpJWXX355t7vnJWrXpRNMUVERJSUlcYchIpJWzGxTMu10iExERCKhBCMiIpFQghERkUgowYiISCSUYEREJBJKMCIiEgklGBERiUSk98GY2UzgB0Am8At3/3Yr7eYAfwBOd/cSM+sO/AI4NYzx1+5+S1t9mtl5wG1AFvAy8Fl3r41yfCIix8Ldqat3auuD33Xu1NW9NV1bX099PdTW17+tXWP7sE3jvLqwj8Y29dTWNem73huna+udem+YrmfeWSPo3zsr0vFGlmDMLBO4EzgfqACWmdlCdy9t1i4HuBp4scnsy4Ae7j7JzHoBpWZ2P7ClpT6B1cB9wAx3X2tmC4BPA7+ManwikpyGjerbNoR1LW1k61vYmL59o/rO+cHGsq6eYONa//aN6ts3suHyd2zUW+mjyfs137DXt7DxbzE5NLyPv9VPKsgwmH3KsPRNMMA0oNzd1wOY2QPApUBps3Y3AbcCX2syz4HeZtYNyAZqgKo2+twFHHH3teH6jwHzUYIR6VA7qw7zWkUlKyr28VpFJSu3VrLnQE3cYQFgBt0yjMwMo1tGRvjbyAh/Zzb5HfxkvGN+98wMenZvmA6XZxqZ1qSPzLfeI8OaTlsL75nR7D3tHTE2X7fhPTKsyTiavMfb3zPjneOy4P07QpQJJp9gj6NBBXBG0wZmNhUodPdFZtY0wTxIkDi2A72Ar7j7XjNrrc/dQHczK3b3EmAOUNhSUGZ2BXAFwPDhw9/F8ES6tr0HalhesY8VFZUs31rJ8op9vFF1BAi+IY8ZnMN5YwcxtF823cMNccNGNdMgM/OdG/DMY9zwNt/oNmxAu2VkNHm/jt2oyluiTDAtfZqN+4dmlgHcAcxrod00oA4YBpwAPGtmj7fWp7u7mc0F7jCzHsDfgBbPv7j73cDdAMXFxamxvyqS4qoOH2Xl1kqWV1SyoqKS1yr2UfHmocblI/N6c+bIAUwu6MfkglzGD+tLr6wu/ahDIdoEU8Hb9yIKgG1NpnOAicBSMwMYAiw0s9nA5cAj7n4U2GlmzwHFBHsvLfbp7s8D7wMwswuAMRGMSaTTO1hTS+m2qsZDXcsrKlm/+0Dj8oITsplS0I9PTD+RyQW5TMzPpW/P7jFGLKkqygSzDBhtZiOArcBcgsQBgLtXAgMbps1sKfC18CqyGcB5ZvY/BIfIpgPfJzh/02KfZjbI3XeGezD/Adwc4dhEOoUjtXWs3r4/OMS1ZR8rtlay9o39NJyLHty3B5ML+vEvU/OZXNiPSfm5kZ8Yls4jsgTj7rVmdhXwKMElxfe4+6rwCq8Sd1/Yxup3AvcCKwkOi93r7ssBWuozXOfrZnYJwb09P3X3JyMZmEiaqq2rZ+0b1azYui/cO6lk9Y4qjtYF2aR/7ywmF+RywfjBTAoPdQ3u2zPmqCWdmXvXPQ1RXFzsqgcjnVF9vbN+dzXLKyrDn32s2lbFkdp6AHJ6dmNSfm7jOZPJBbnk98smPFwt0iYze9ndixO101k4kTTn7mzZe4jXKoJDXK9tCZJJ9ZHgOpfs7plMzO/beM5kckE/TuzfS1dVSeSUYETSiLuzo+owr22pZMXW4AT8iq2V7Dt4FICszAzGDevLh07Nb9xDGTWoD5lKJhIDJRiRFLa7+kjjZcHB70p2Vwf3mmRmGCcPzmHmhCGNh7rGDM4hq5seMSipQQlGJEVUHjwaHOJquHmxYh/bKg8DwV3oo/L6cPaYgUwp6MekglzGD+1Lz+6ZMUct0jolGJEYVB+pZVV44+LyrcH9Jhv3HGxcXjSgF6cV9ecz+cEJ+An5ufTpoT9XSS/6HysSscNH6yjdXvW2Q13lu6ppuIBzWG5PJhf047LiwmDvJD+X3F66cVHSnxKMSDs6WlfPmh37Gy8NXl4R3LhYG965OLBPD6YU5HLx5KFMKejHxPxc8nJ6xBy1SDSUYESOU129U76zOnjg49bgBHzZ9ipqwntNcrO7M7kglyvHjmRSfj+mFOYypG9P3WsiXYYSjEgS6uudTXsPNu6VLK/Yx8qtVRw6WgdA76xMJubnMu89RUzKz2VKQT8K++vGRenalGBEmnF3tu471HgXfMP9JvsPBzcu9uiWwYRhffno6YWNd8GPHNhHNy6KNKMEI13ezqrDb50z2Ro8o6uhSFb3TGPskL7MmjKMKQW5TMrvx5jBfeiWqXtNRBJRgpEu5c0DNY1PDm5IJjuqgntNmhbJmlzYj8n5uYwdmkOPbrrXROR4KMFIp9VQJGtFwwMft+5jy963F8maPrI/kwr6MUVFskTanf6apNNYUVHJso17Gw91rd/1ziJZHz9DRbJEOooSjHQKD722jS/e/0+gSZGsU1QkSyROSjCS9g7W1PLfS8qYmN+XX376dBXJEkkRSjCS9n729Hq2Vx7mRx+bquQikkJ0raWktYo3D3LX0+uYPWUYxUX94w5HRJpQgpG0dsuS1ZjBtReOjTsUEWlGCUbS1gvr97B4xXa+cM4ohvXLjjscEWlGCUbSUl29c+NDpeT3y+aKs0fGHY6ItEAJRtLS75ZtoWx7FfMvGkt2lu60F0lFSjCSdioPHeW2v61h2oj+XDxpaNzhiEgrlGAk7fzwidd582ANN8war8fhi6QwJRhJK+U7q7nvHxuZe/pwJgzLjTscEWmDEoyklW8tLiU7K5OvXTAm7lBEJAElGEkbT63eydI1u/jSjNEM6KM69iKpTglG0kJNbT03LSplZF5vPnVmUdzhiEgSIk0wZjbTzNaYWbmZXdtGuzlm5mZWHE53N7P7zGyFmZWZ2fxEfZrZDDN7xcxeNbO/m9moKMcmHeu+f2xk/e4DfPOS8WR10/cikXQQ2V+qmWUCdwIXAuOBj5nZ+Bba5QBXAy82mX0Z0MPdJwGnAVeaWVGCPn8KfNzdTwH+F7gumpFJR9u1/wg/fOJ13n9yHu8/eVDc4YhIkqL8KjgNKHf39e5eAzwAXNpCu5uAW4HDTeY50NvMugHZQA1QlaBPB/qGr3OBbe08HonJ9/62hkNH67juknd8PxGRFBZlgskHtjSZrgjnNTKzqUChuy9qtu6DwAFgO7AZuM3d9ybo83PAEjOrAD4JfLudxiExWrm1kt+VbGHee4o4Ka9P3OGIyDGIMsG0dAecNy40ywDuAK5pod00oA4YBowArjGzkQn6/ApwkbsXAPcCt7cYlNkVZlZiZiW7du1KdiwSA3fnxodW0b9XFl+cMTrucETkGEWZYCqAwibTBbz9sFUOMBFYamYbgenAwvBE/+XAI+5+1N13As8Bxa31aWZ5wBR3bziP8zvgPS0F5e53u3uxuxfn5eW92zFKhBYt386yjW/y9Q+eTG5297jDEZFjFGWCWQaMNrMRZpYFzAUWNix090p3H+juRe5eBLwAzHb3EoLDYudZoDdB8lndRp9vArlm1nD33flAWYRjk4gdqqnjliVlTBjWl8uKCxOvICIpJ7KSye5ea2ZXAY8CmcA97r7KzBYAJe6+sI3V7yQ4zLWS4LDYve6+HKClPsP5/wb80czqCRLOv0Y0NOkAdz2zjm2Vh/n+3KlkZuh5YyLpyNw9catOqri42EtKSuIOQ5rZuu8QM763lA+MG8yPLz817nBEpBkze9ndixO10x1rknK+/fBq3GH+RePiDkVE3gUlGEkpL23Yy0OvbePz55xEvsogi6Q1JRhJGUEZ5FUMze3J5885Ke5wRORdUoKRlPGHki2s2lbF/IvGqQyySCegBCMpoerwUb776BqKTzyBWZNVBlmkM4jsMmWRY/GjJ15n78EafjVrmsogi3QS2oOR2K3bVc29z23kI6cVMqlAZZBFOgslGIndzYvLyO6eydc+eHLcoYhIO1KCkVg9tWYnT67eydUzRpOXozLIIp2JEozE5mhdUAZ5xMDefPo9RXGHIyLtTAlGYvPr5zexftcBvnnJOJVBFumE9FctsdhTfYTvP76Wc8aoDLJIZ6UEI7H43mNrOVRTxzcvGafLkkU6KSUY6XCrtlVy/0ub+dSZRYwalBN3OCISESUY6VBBGeRSTuiVxZdUBlmkU1OCkQ61ZMUOXtqwl2suGENuL5VBFunMlGCkwxw+Wsd/Lylj7JAc5p4+PO5wRCRiehaZdJi7n1nP1n2HeOCK6SqDLNIFaA9GOsS2fYf4ydJyLp40lOkjB8Qdjoh0ACUY6RDfeSQog3zthWPjDkVEOogSjESuZONe/vrqNq48eySF/XvFHY6IdBAlGIlUfX1wWfKQvj35/LkqgyzSlSjBSKQefKWCFVsrmX/RWHpl6ZoSka5ECUYis//wUW59ZA2nnXgCs6cMizscEelgSjASmR8/Wc7u6iPcMGu8njcm0gUpwUgkNuw+wD3PbeCy0wqYXNAv7nBEJAZKMBKJmxeXkpWZwddnqgyySFelBCPt7um1u3i8bCdfnDGaQTk94w5HRGKSVIIxs7PMrHf4+hNmdruZnRhtaJKOGsogFw3oxWfOKoo7HBGJUbJ7MD8FDprZFOAbwCbg14lWMrOZZrbGzMrN7No22s0xMzez4nC6u5ndZ2YrzKzMzOYn6tPMnjWzV8OfbWb2lyTHJu3of17YRPnOaq67eDw9umXGHY6IxCjZBFPr7g5cCvzA3X8AtFkpyswygTuBC4HxwMfMbHwL7XKAq4EXm8y+DOjh7pOA04ArzayorT7d/X3ufoq7nwI8D/wpybFJO9l7oIY7HlvL+0YPZMY4lUEW6eqSTTD7w72ITwCLww19omIe04Byd1/v7jXAAwQJqrmbgFuBw03mOdDbzLoB2UANUJVMn2HCOg/QHkwHu/2xNRyoqeP6S3RZsogkn2A+ChwBPuvuO4B84LsJ1skHtjSZrgjnNTKzqUChuy9qtu6DwAFgO7AZuM3d9ybTJ/AvwBPuXpVoUNJ+yrZX8b8vbuaT009k9GCVQRaR5OvB7Cc4NFZnZmOAscD9CdZp6SusNy40ywDuAOa10G4aUAcMA04AnjWzxxP1GfoY8ItWgzK7ArgCYPhwFb1qD+7OgodKyc3uzlc+MCbucEQkRSS7B/MM0MPM8oEngM8Av0qwTgVQ2GS6ANjWZDoHmAgsNbONwHRgYXii/3LgEXc/6u47geeA4kR9mtkAguS0uLWg3P1udy929+K8vLwEQ5BkPLpqB8+v38NXLzhZZZBFpFGyCcbc/SDwIeBH7v4vwIQE6ywDRpvZCDPLAuYCCxsWunuluw909yJ3LwJeAGa7ewnBYbHzLNCbIPmsTtQnwcUBi9y96fkcidDho3V8a3FQBvljpxcmXkFEuoykE4yZnQl8nLf2Dtq8BtXda4GrgEeBMuD37r7KzBaY2ewE73cn0AdYSZBU7nX35a312WS9uSQ+dCft6BfPrqfizUNcf8l4umXqvl0ReUuy52C+DMwH/hwmiZHAU4lWcvclwJJm865vpe25TV5XE+yNJNVnS31I9HZUHubOp9Yxc8IQ3jNqYNzhiEiKSSrBuPvTwNNmlmNmfdx9PcG9K9KFfeeR1dS5858Xj4s7FBFJQck+KmaSmf2T4JBVqZm9bGaJzsFIJ/bypjf58z+3csX7VAZZRFqW7EHzu4CvuvuJ7j4cuAb4eXRhSSqrr3cWPLSKwX178AWVQRaRViSbYHq7e+M5F3dfCvSOJCJJeX/651Zeq6jk2gvH0ruHyiCLSMuS3TqsN7NvAr8Jpz8BbIgmJEll1Udq+c4jq5k6vB+XTmn+EAURkbckuwfzr0AewQMk/xy+/kxUQUnquvOpcnbtP8INsyaQkaHnjYlI65K9iuxNdNVYl7dpzwF++ewGPnxqAacUqgyyiLStzQRjZg/xzmd9NXL3RDdMSidy8+Iyumca/6EyyCKShER7MLd1SBSS8v7++m7+VvoG35h5MoP6qgyyiCTWZoIJb7CULq62rp4bH1rF8P69+NezRsQdjoikiaTOwZjZCt55qKwSKAG+5e572jswSR2/fXEzr++s5q5PnkbP7iqDLCLJSfYy5YcJ6rP8bzg9l6A2SyXBY/tntXtkkhLePFDD7Y+t5axRA7hg/OC4wxGRNJJsgjnL3c9qMr3CzJ5z97PM7BNRBCap4Y7H11J9pJbrL5mgMsgickySvQ+mj5md0TBhZtMIHqcPUNvuUUlKWL2jiv95YROfOGM4Jw9RGWQROTbJ7sF8DrjHzPoQHBqrAj4bFgO7JargJD4NZZD7ZnfnK+erDLKIHLtkb7RcBkwys1yC6pb7miz+fSSRSaz+VvoG/1i3hwWXTqBfr6y4wxGRNJTs4/pzzex24AngcTP7XphspBM6fLSOmxeXMWZwHy6fNjzucEQkTSV7DuYeYD/wkfCnCrg3qqAkXvc8t4HNew9yw6wJKoMsIsct2XMwJ7n7h5tM32hmr0YRkMTrjarD/PjJci4YP5izVAZZRN6FZL+eHjKz9zZMmNlZwKFoQpI4feeR1dTWqQyyiLx7ye7BfAG4r+EkP7AXmBdVUBKPf25+kz+9spUvnHsSJw5QPTkReXeSvYrsVWCKmfUNp6sijUo6XH29c+NDpeTl9ODf3z8q7nBEpBNI9Lj+r7YyHwB3vz2CmCQGf3l1K69u2cdtl02hj8ogi0g7SLQl0e3bXcCBI7V8++HVTCnsx4emqgyyiLSPRI/rv7GjApH4/GRpOTv3H+FnnzxNZZBFpN0c800OZvZKFIFIPDbvOcjPn93Ah6bmc+rwE+IOR0Q6keO5i05fcTuR/15SRrcM4xszx8Ydioh0MseTYBa3exQSi3+U7+aRVTv49/ePYkiuyiCLSPs65gTj7tdFEYh0rNq6ehYsKqXghGw++16VQRaR9pfswy73m1lVs58tZvZnMxvZxnozzWyNmZWb2bVttJtjZm5mxeF0dzO7z8xWmFmZmc1P1KcFbjazteE6Vyf3T9A13b9sC6t37Oe6i8epDLKIRCLZGx5uB7YRlEw2gpLJQ4A1BA/CPLf5CmaWCdwJnA9UAMvMbKG7lzZrlwNcDbzYZPZlQA93n2RmvYBSM7sf2NJGn/OAQmCsu9eb2aAkx9bl7DtYw/f+toYzRw7ggxOGxB2OiHRSyR4im+nud7n7fnevcve7gYvc/XdAa5ceTQPK3X29u9cADwCXttDuJuBW4HCTeQ70NrNuQDZQQ/AE57b6/AKwwN3rAdx9Z5Jj63K+//jrVB06yvWzxqsMsohEJtkEU29mHzGzjPDnI02WeSvr5BPscTSoCOc1MrOpQKG7L2q27oPAAWA7sBm4zd33JujzJOCjZlZiZg+b2eiWgjKzK8I2Jbt27Wp1wJ3V2jf285sXNnH5GcMZN7Rv3OGISCeWbIL5OPBJYCfwRvj6E2aWDVzVyjotfTVuTEZmlgHcAVzTQrtpQB0wDBgBXBOe62mrzx7AYXcvBn5OcOjunY3d73b3YncvzsvLayX0zsnduWlRKb2zMvnq+SfHHY6IdHLJPuxyPTCrlcV/b2V+BcE5kQYFBOdxGuQAE4Gl4WGaIcBCM5sNXA484u5HgZ1m9hxQTLD30lqfFcAfw9d/RgXR3uHxsp08+/pu/mvWePr3VhlkEYlWsleRjTGzJ8xsZTg92cwSXa68DBhtZiPMLIvgwoCFDQvdvdLdB7p7kbsXAS8As929hOCw2HnhlWG9genA6gR9/gU4L3x9DrA2mbF1FUdq6/jW4lJGD+rDx6efGHc4ItIFJHuI7OfAfOAogLsvJ9i4t8rdawkOnz0KlAG/d/dVZrYg3Etpy51AH2AlQVK5192Xt9ZnuM63gQ+b2QrgFuBzSY6tS7j3uY1s2nOQ62eNp7vKIItIB0j2MuVe7v5SsyuOahOt5O5LgCXN5l3fSttzm7yuJrhUOak+w/n7gIsTxdQV7dx/mB898TofGDeY943uWuedRCQ+yX6V3W1mJxGeUDezOQRXeEka+O4ja6ipq+c6lUEWkQ6U7B7MvwN3A2PNbCuwgeDKMklxr23Zxx9eruDKc0ZSNFBlkEWk4ySbYLYSXJX1FNCf4KbHTwMLIopL2oG7c+NDqxjYpwdXqQyyiHSwZBPMX4F9wCu8/VJjSWF/fXUbr2zex61zJpPTs3vc4YhIF5Nsgilw95mRRiLt6sCRWm55uIxJ+bnMObUg7nBEpAtK9iT/P8xsUqSRSLv62dPreKPqCP81e7zKIItILJLdg3kvMM/MNgBHCB7Z4u4+ObLI5Lht2XuQu55Zz6WnDOO0E/vHHY6IdFHJJpgLI41C2tUtD5eRaca1F6oMsojEJ9lnkW2KOhBpH8+v28OSFTu45vwxDM3NjjscEenC9MyQTqSuPrgsOb9fNv92dquFRkVEOoQSTCfywLLNrN6xn/9UGWQRSQFKMJ1E5cGj3PboGs4Y0Z8LJ6oMsojETwmmk/jBE69TqTLIIpJClGA6gfKd+/n18xuZO204E4blxh2OiAigBJP23J0Fi8rIzsrkmvPHxB2OiEgjJZg09+TqnTyzdhdf/sAYBvTpEXc4IiKNlGDSWE1tPTctKuWkvN586kyVQRaR1KIEk8Z+9Y8NbNxzkOtnTVAZZBFJOdoqpald+4/wwyfKmTF2EOeMURlkEUk9SjBp6rZH13Ckto7/VBlkEUlRSjBpaEVFJb9/eQufOWsEI/P6xB2OiEiLlGDSTEMZ5AG9s7jqPJVBFpHUpQSTZh5avp2STW/y9Q+eTF+VQRaRFKYEk0YO1dRxy5IyJub3Zc5phXGHIyLSJiWYNPKzp9exvfIwN8yaQKbKIItIilOCSRMVbx7kZ0+vY9aUYZxepDLIIpL6lGDSxC0Pr8YMlUEWkbShBJMGXly/h8XLt/P5c04iv5/KIItIeog0wZjZTDNbY2blZnZtG+3mmJmbWXE43d3M7jOzFWZWZmbzE/VpZr8ysw1m9mr4c0qUY+soQRnkUvL7ZXPl2SfFHY6ISNK6RdWxmWUCdwLnAxXAMjNb6O6lzdrlAFcDLzaZfRnQw90nmVkvoNTM7ge2JOjz6+7+YFRjisPvS7ZQur2KH18+lewslUEWkfQR5R7MNKDc3de7ew3wAHBpC+1uAm4FDjeZ50BvM+sGZAM1QNUx9NkpVB4KyiBPK+rPxZOGxh2OiMgxiTLB5BPscTSoCOc1MrOpQKG7L2q27oPAAWA7sBm4zd33JtHnzWa23MzuMLO0L47yoydeZ+/BGpVBFpG0FGWCaWmL6I0LzTKAO4BrWmg3DagDhgEjgGvMbGSCPucDY4HTgf7Af7QYlNkVZlZiZiW7du1Kcigdb92uan71j43MPb2Qifkqgywi6SfKBFMBNL3dvADY1mQ6B5gILDWzjcB0YGF4ov9y4BF3P+ruO4HngOK2+nT37R44AtxLkKTewd3vdvdidy/Oy0vdx9x/a1Ep2d0zueaCk+MORUTkuESZYJYBo81shJllAXOBhQ0L3b3S3Qe6e5G7FwEvALPdvYTgsNh5FuhNkHxWt9WnmQ0Nfxvwf4CVEY4tUk+t3slTa3bxpQ+MZqDKIItImorsKjJ3rzWzq4BHgUzgHndfZWYLgBJ3X9jG6ncS7IWsJDgsdq+7Lwdoqc9wnd+aWV7Y/lXg81GMK2o1tfXctLiUkQN786kzi+IOR0TkuEWWYADcfQmwpNm861tpe26T19UElyon1Wc4/7x3E2uq+PXzG1m/6wD3zjudrG66D1ZE0pe2YClkd/URfvD465x7ch7vHzso7nBERN4VJZgU8r2/reHQ0Tquu3h83KGIiLxrSjApYuXWSh5YtoV57yli1CCVQRaR9KcEkwLcnQUPldK/VxZfnDE67nBERNqFEkwKWLxiOy9t3MvXPngyudkqgywinYMSTMyCMsirGT+0Lx8pVhlkEek8lGBidvcz69m67xA3zBqvMsgi0qkowcRo275D/PTpci6ePJQzRg6IOxwRkXalBBOjbz+8GneYrzLIItIJKcHEZNnGvSx8bRtXnnMSBSf0ijscEZF2pwQTg6AM8iqG5vbk8+eMjDscEZFIKMHE4MGXt7ByaxXXXjiWXlmRPg5ORCQ2SjAdrOrwUb776BqKTzyB2VOGxR2OiEhklGA62I+fLGfPgRpumDVBZZBFpFNTgulA63dVc+9zG/jIaYVMKlAZZBHp3JRgOtDNi8vo0S2Tr31QZZBFpPNTgukgS9fs5InVO7l6xijyclQGWUQ6PyWYDnC0rp6bFpUyYmBv5r1nRNzhiIh0CCWYDvCb5zexbtcBrrt4nMogi0iXoa1dxPZUH+GOx9dy9pg8zlMZZBHpQpRgInb7Y2s5WFPH9ZeM02XJItKlKMFEqHRbFfe/tJlPnXkiowblxB2OiEiHUoKJiHvwvLHc7O58ecaYuMMREelwSjAReXjlDl7cEJZB7qUyyCLS9SjBRODw0TpuXlzG2CE5zD19eNzhiIjEQo/yjcDPwzLI9//bdJVBFpEuS3sw7Wx75SF+snQdF00awpknqQyyiHRdSjDt7DsPr6bOnfkXjos7FBGRWCnBtKOXN+3lL69u48qzR1LYX2WQRaRrizTBmNlMM1tjZuVmdm0b7eaYmZtZcTjd3czuM7MVZlZmZvOT7dPMfmRm1dGMqHX19c6ND5UypG9PvnDuSR399iIiKSeyBGNmmcCdwIXAeOBjZja+hXY5wNXAi01mXwb0cPdJwGnAlWZWlKjPMEH1i2hIbfrjKxUsr6hUGWQRkVCUezDTgHJ3X+/uNcADwKUttLuf0e9SAAAIPklEQVQJuBU43GSeA73NrBuQDdQAVW31GSaf7wLfiGg8rdp/+CjfeWQNpw7vx6WnqAyyiAhEm2DygS1NpivCeY3MbCpQ6O6Lmq37IHAA2A5sBm5z970J+rwKWOju29ttBEn68VPl7K4+ojLIIiJNRHksp6UtrTcuNMsA7gDmtdBuGlAHDANOAJ41s8db69PMhhEcVjs3YVBmVwBXAAwf/u5vgtyw+wD3/H0Dc04rYEphLEfnRERSUpR7MBVAYZPpAmBbk+kcYCKw1Mw2AtOBheF5lMuBR9z9qLvvBJ4DitvocyowCigP++plZuUtBeXud7t7sbsX5+XlvetB3ry4jKzMDL4xU2WQRUSaijLBLANGm9kIM8sC5gILGxa6e6W7D3T3IncvAl4AZrt7CcFhsfMs0Jsg+axurU93X+zuQ5r0ddDdR0U4NgCeWbuLx8ve4IszRjMop2fUbyciklYiSzDuXktwXuRRoAz4vbuvMrMFZjY7wep3An2AlQRJ5V53X95an1GNoS0NZZBPHNCLz5xVFEcIIiIpLdLrad19CbCk2bzrW2l7bpPX1QTnVJLqs4U2fY411mP12xc28frOan7+qWJ6dMuM+u1ERNKO7uQ/DnsP1HD7Y2t53+iBfGCcyiCLiLRECeY43PHYWg7U1PHNS8brsmQRkVYowRyHghOyueLskYwZrDLIIiKt0TNNjsOV5+hZYyIiiWgPRkREIqEEIyIikVCCERGRSCjBiIhIJJRgREQkEkowIiISCSUYERGJhBKMiIhEwtw9catOysx2AZuOc/WBwO52DCdOnWUsnWUcoLGkqs4ylnc7jhPdPWFBrS6dYN4NMytx9+K442gPnWUsnWUcoLGkqs4ylo4ahw6RiYhIJJRgREQkEkowx+/uuANoR51lLJ1lHKCxpKrOMpYOGYfOwYiISCS0ByMiIpFQgknAzGaa2RozKzeza1tY3sPMfhcuf9HMijo+ysSSGMc8M9tlZq+GP5+LI85kmNk9ZrbTzFa2stzM7IfhWJeb2akdHWMykhjHuWZW2eQzub6jY0yWmRWa2VNmVmZmq8zsSy20SfnPJclxpMXnYmY9zewlM3stHMuNLbSJdvvl7vpp5QfIBNYBI4Es4DVgfLM2/xf4Wfh6LvC7uOM+znHMA34cd6xJjuds4FRgZSvLLwIeBgyYDrwYd8zHOY5zgUVxx5nkWIYCp4avc4C1LfwfS/nPJclxpMXnEv479wlfdwdeBKY3axPp9kt7MG2bBpS7+3p3rwEeAC5t1uZS4L7w9YPADDOzDowxGcmMI224+zPA3jaaXAr82gMvAP3MbGjHRJe8JMaRNtx9u7u/Er7eD5QB+c2apfznkuQ40kL471wdTnYPf5qfdI90+6UE07Z8YEuT6Qre+Z+tsY271wKVwIAOiS55yYwD4MPhoYsHzaywY0KLRLLjTQdnhoc4HjazCXEHk4zwMMtUgm/MTaXV59LGOCBNPhczyzSzV4GdwGPu3upnEsX2SwmmbS1l8ubfAJJpE7dkYnwIKHL3ycDjvPWtJh2lw2eSjFcIHskxBfgR8JeY40nIzPoAfwS+7O5VzRe3sEpKfi4JxpE2n4u717n7KUABMM3MJjZrEulnogTTtgqg6Tf5AmBba23MrBuQS+od9kg4Dnff4+5HwsmfA6d1UGxRSOZzS3nuXtVwiMPdlwDdzWxgzGG1ysy6E2yUf+vuf2qhSVp8LonGkW6fC4C77wOWAjObLYp0+6UE07ZlwGgzG2FmWQQnwRY2a7MQ+HT4eg7wpIdnzFJIwnE0OxY+m+DYc7paCHwqvGppOlDp7tvjDupYmdmQhuPhZjaN4O91T7xRtSyM85dAmbvf3kqzlP9ckhlHunwuZpZnZv3C19nAB4DVzZpFuv3q1l4ddUbuXmtmVwGPElyJdY+7rzKzBUCJuy8k+M/4GzMrJ8j8c+OLuGVJjuNqM5sN1BKMY15sASdgZvcTXMkz0MwqgBsITmDi7j8DlhBcsVQOHAQ+E0+kbUtiHHOAL5hZLXAImJuCX14anAV8ElgRHvMH+H/AcEirzyWZcaTL5zIUuM/MMgmS4O/dfVFHbr90J7+IiERCh8hERCQSSjAiIhIJJRgREYmEEoyIiERCCUZERCKhBCMSITOrTtyqzfUfNLOR4es+ZnaXma0Ln477jJmdYWZZ4WvddiApRQlGJEWFz7jKdPf14axfENyrMNrdJxDcqzQwfIDpE8BHYwlUpBVKMCIdILx7/btmttLMVpjZR8P5GWb2k3CPZJGZLTGzOeFqHwf+GrY7CTgDuM7d6wHCp2MvDtv+JWwvkjK0Sy3SMT4EnAJMAQYCy8zsGYI7x4uAScAggkf03BOucxZwf/h6AvCqu9e10v9K4PRIIhc5TtqDEekY7wXuD59u+wbwNEFCeC/wB3evd/cdwFNN1hkK7Eqm8zDx1JhZTjvHLXLclGBEOkZrRZzaKu50COgZvl4FTDGztv5mewCHjyM2kUgowYh0jGeAj4YFoPIIyiW/BPydoNBbhpkNJnj4ZYMyYBSAu68DSoAbmzzJd7SZXRq+HgDscvejHTUgkUSUYEQ6xp+B5cBrwJPAN8JDYn8kqMmxEriLoHpiZbjOYt6ecD4HDAHKzWwFQd2ehnoq7yd4WrFIytDTlEViZmZ93L063At5CTjL3XeENTyeCqdbO7nf0MefgPnuvqYDQhZJiq4iE4nforAwVBZwU7hng7sfMrMbCOqmb25t5bCI3F+UXCTVaA9GREQioXMwIiISCSUYERGJhBKMiIhEQglGREQioQQjIiKRUIIREZFI/H87YhPVU56zcgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# scores_：dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold,\n",
    "# Each dict value has shape (n_folds, len(Cs))\n",
    "n_Cs = len(Cs)    \n",
    "mse_mean = -np.mean(lrcv_L1.scores_[1],axis = 0)\n",
    "pyplot.plot(np.log10(Cs), mse_mean.reshape(n_Cs,1)) \n",
    "#plt.plot(np.log10(reg.Cs)*np.ones(3), [0.28, 0.29, 0.30])\n",
    "pyplot.xlabel('log(C)')\n",
    "pyplot.ylabel('neg-logloss')\n",
    "pyplot.show()\n",
    "\n",
    "#print ('C is:',lr_cv.C_)  #对多类分类问题，每个类别的分类器有一个C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "log(C) = 0时，即C = 1时，Jcv(cross validation error)最小"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.41997772,  1.11869971, -0.23277117,  0.03897164, -0.10281628,\n",
       "         0.72279967,  0.2081879 ,  0.133444  ]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lrcv_L1.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "惩罚不够，没有稀疏系数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测\n",
    "评价带L1正则项的LogisticRegressionCV预测性能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the correct classification fraction of the zero_one_loss of LogisticRegressionCV on test is 0.7402597402597403\n",
      "the correct classification fraction of the zero_one_loss of LogisticRegressionCV on train is 0.7817589576547231\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import zero_one_loss  #评价逻辑回归预测模型的性能\n",
    "# 预测\n",
    "y_test_pred_lr = lrcv_L1.predict(X_test)\n",
    "y_train_pred_lr = lrcv_L1.predict(X_train)\n",
    "# 使用zero_one_loss评价模型在测试集和训练集上的性能，并输出评估结果\n",
    "#测试集\n",
    "print ('the correct classification fraction of the zero_one_loss of LogisticRegressionCV on test is', 1 - zero_one_loss(y_test, y_test_pred_lr))\n",
    "#训练集\n",
    "print ('the correct classification fraction of the zero_one_loss of LogisticRegressionCV on train is', 1 - zero_one_loss(y_train, y_train_pred_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用带L1正则项的LogisticRegressionCV做预测，测试集上正确分类的比例是0.7402597402597403，训练集上正确分类的比例是0.7817589576547231"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### L2正则"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegressionCV(Cs=[1, 10, 100, 1000], class_weight=None, cv=5,\n",
       "           dual=False, fit_intercept=True, intercept_scaling=1.0,\n",
       "           max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2',\n",
       "           random_state=None, refit=True, scoring='neg_log_loss',\n",
       "           solver='liblinear', tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "Cs = [1, 10,100,1000]\n",
    "\n",
    "# 为了和GridSeachCV比较，也用liblinear\n",
    "\n",
    "lr_cv_L2 = LogisticRegressionCV(Cs=Cs, cv = 5, scoring='neg_log_loss', penalty='l2', solver='liblinear', multi_class='ovr')\n",
    "lr_cv_L2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{1: array([[-0.46129644, -0.46088502, -0.4608487 , -0.46084513],\n",
       "        [-0.46510588, -0.46473286, -0.46469927, -0.46469595],\n",
       "        [-0.56426612, -0.56834734, -0.56879096, -0.5688357 ],\n",
       "        [-0.44377717, -0.44400066, -0.4440332 , -0.44403656],\n",
       "        [-0.46617809, -0.46612355, -0.4661268 , -0.46612722]])}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_cv_L2.scores_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xucl3Wd9/HXe4bhPIIIijDgoIgCoqkTWZoaWZ5xt7TQ6s6s2+pe17uDHdxKU+uudU23u2zLWlndbbOWdg0BtTyleasxtsgZRUQZREGF4cycPvcf1wWOODO/HzDXXL8Z3s/HYx78ruv3vb58vvyG6/O7Tt+PIgIzM7POVpZ3AGZm1jM5wZiZWSacYMzMLBNOMGZmlgknGDMzy4QTjJmZZcIJxszMMuEEY2ZmmXCCMTOzTPTKO4A8DR06NKqrq/MOw8ysW3n66adfi4hhhdrt1wmmurqa2travMMwM+tWJL1YTDufIjMzs0w4wZiZWSacYMzMLBNOMGZmlgknGDMzy4QTjJmZZcIJxszMMpHpczCSzgJ+CJQDv4iI77fT7kLgP4B3RkStpArgF8AJaYx3RsT3OupT0hTgJqA38DTw6YhoynJ8ZmadLSJobgmaWoKG5haamoOm5pY3X7e00NCU/NnYHDSm6xtb0j+bW95c19xCY0uyfbI+dvXxqZPHMGRA70zHklmCkVQO3Ap8AKgD5kqaGRGLd2tXCVwJPNVq9UVAn4iYJKk/sFjSr4BVbfUJLAXuAN4fEc9Kuh74JPDPWY3PzEpTRCQ70t13wM0tNLW0sQPe1fatO+CGpqR9snNP/mxqiXR9sn3rnf7uf9db/76dO/ni2mZNggveMaL7JhhgMrA8IlYASLoLuABYvFu7G4AbgatarQtggKReQD+gAdjYQZ/rgB0R8Wy6/R+Aq3GCMctFRFC3fhvz6+p5bfOON3egTW18o25pobHpzW/grb+hv7kTb71zf3Nn3VbSaGrJfgddXiZ6lYmK8jIqykWv8jIqykRFr7JW68voVS4qysroW1FGrz69krZlZVT0Str3Km/VNt0+WV/2Zt/pcu+0v+R10k+vcqXrW79O/460750xtI6tvEyZ/xtBtglmJMkRx051wLtaN5B0PDAqImZJap1gZpAkjjVAf+CLEfGGpPb6fA2okFQTEbXAhcCotoKSdDlwOcDo0aP3YXhmBkkyeWXjdubX1bOgrp75q+tZULeB9Vsb292mV1s719Y761Y714ryMvr37rVrx9m71+47153ry1rtjJM/37az37nT3X0HvFscb+t718492VmXddEOurvLMsG09Qns+mohqQy4Bbi0jXaTgWZgBHAg8JikB9rrMyJC0jTgFkl9gN8DbV5/iYjbgNsAampqsv+qY9bDrNu0g/l1G5KEsrp+11EKJN/sxx1SyQcnDGdS1SCOqxrMoYP77koYO3fiknfQ+4MsE0wdbz2KqAJebrVcCRwDPJL+sg0HZkqaClwC3BcRjcBaSY8DNSRHL232GRFPAO8FkPRBYFwGYzLbr6zf0rDriGRnQllTvx1IzuOPHTaQ08YN49iqQUyqGsSEQw+gb0V5zlFbqcgywcwFjpQ0BlgNTCNJHABERD0wdOeypEeAq9K7yN4PTJH0bySnyE4C/pHk+k2bfUo6OCLWpkcwXwO+m+HYzHqc+m2NLFq98xRXPfNXb2DVG9t2vX/40AFMHjOESSMHcWzVYCaOOIABffbrCdmtgMx+OyKiSdIVwP0ktxTfHhGL0ju8aiNiZgeb3wpMBxaSnBabHhHzAdrqM93mK5LOI3m2558i4qFMBmbWA2zZ0cSilze+5VTXC69t2fX+qCH9OHbkYD72rsM4duQgjqkaxAF9K3KM2LojRey/lyFqamrC9WCsp9ve2MziNRuZv2rDrqOT5es2s/O//qGD+jJp5CCOGzWYSSMHMWnkIA7M+PZV694kPR0RNYXa+fjWrAfZ0dTMslc2veWOrmdf3URzeuvu0IF9OK5qEOceeyjHVg3imJGDOLiyb85RW0/lBGPWTTU2t/Dcq5tZsPrN01xL12yiobkFgAP7VzCpajDvP/pgJlUN4tiqQQw/oK/v4LIu4wRj1g00twQr1m3elUieqdvA4pc3sqMpSSaVfXsxaeQgPnVKNcdVJae6qg7s52RiuXKCMSsxLS3Bi29sffMCfF09C1+uZ2tDMwD9e5dzzIhBfOKkw9Ijk8EcNqS/H/6zkuMEY5aj1lOqzF+9gQXpEcqm7clzwn16lTFxxAF8pGZUenvwIA4fNrDLpvow2xdOMGZdpNCUKhXlYvyhBzD1uBHJg4sjB3PkIQOpKHdVDeuenGDMMrJu045dF+B3/nQ0pcq44QPp08tPwVvP4QRj1gnWb2lI5+XylCpmOznBmO2hjdsbWVjnKVXMCvFvvVkHWk+psiBNKCs8pYpZUZxgzFI7p1RZsOuayYY2p1T58IlVnlLFrAhOMLZfamhqYdkrm3imboOnVDHLiBOM9XhNzS086ylVzLqcE4z1KLtPqTK/bgOLPKWKWS6cYKxHeKV+O1+Z8QxPv7jeU6qYlQgnGOsRbpi9mD+/8AYXTx7tKVXMSkSmc1BIOkvSMknLJX29g3YXSgpJNelyhaQ7JC2QtETS1YX6lPR+SX+RNE/SnySNzXJsVjqeWvE6s+ev4XOnHcG3p07kwydWceQhlU4uZjnLLMFIKicpfXw2MAG4WNKENtpVAlcCT7VafRHQJyImAScCn5VUXaDPfwI+FhHvAP4d+GY2I7NS0twSXHfPYkYM6svnTjsi73DMrJUsj2AmA8sjYkVENAB3ARe00e4G4EZge6t1AQyQ1AvoBzQAGwv0GcAB6etBwMudPB4rQb+eu4rFazZy9Tnj6dfbU6+YlZIsE8xIYFWr5bp03S6SjgdGRcSs3badAWwB1gAvATdFxBsF+vwMMEdSHfAJ4PudNA4rUfXbGrnp98uYXD2E8449NO9wzGw3WSaYtk6Ax643pTLgFuDLbbSbDDQDI4AxwJclHV6gzy8C50REFTAduLnNoKTLJdVKql23bl2xY7ES9MMHnmP91gauOX+CbzM2K0FZJpg6YFSr5SreetqqEjgGeETSSuAkYGZ6of8S4L6IaIyItcDjQE17fUoaBhwXETuv4/waeE9bQUXEbRFRExE1w4YN29cxWk6Wr93EnU+sZNo7R3PMyEF5h2NmbcgywcwFjpQ0RlJvYBowc+ebEVEfEUMjojoiqoEngakRUUtyWmyKEgNIks/SDvpcDwySNC7t/gPAkgzHZjmKCK6ftYR+vcu56oPjCm9gZrnI7DmYiGiSdAVwP1AO3B4RiyRdD9RGxMwONr+V5DTXQpLTYtMjYj5AW32m6/8n8FtJLSQJ57KMhmY5e2jpWh59dh3fOm8CBw3sk3c4ZtYORUThVj1UTU1N1NbW5h2G7YEdTc2cecujlJeJ+75wqssJm+VA0tMRUVOonf93WrfyL4+vZOXrW/nWeROcXMxKnP+HWrexdtN2fvTQct5/9MGcftTBeYdjZgU4wVi38Q/3LWNHUzPfPO9tE0KYWQlygrFu4ZlVG/iPp+u47OQxjBk6IO9wzKwITjBW8iKCb9+ziKED+3DFFM9hatZdOMFYybt73mr++6UNfPWso6jsW5F3OGZWJCcYK2lbdjTx/XuXcmzVIC48oSrvcMxsDzjBWEn7ySPLeXXjDq49f6IrUZp1M04wVrJeen0rP3/sBf7qHSM48bAD8w7HzPaQE4yVrO/OWUy5xNfPHp93KGa2F5xgrCQ9vvw17l/0Kn/zviMYPqhv3uGY2V5wgrGS09TcwvX3LKbqwH585r2H5x2Ome0lJxgrOf/+55dY9uomvnnuePpWuAyyWXflBGMlZf2WBn7w+2d5zxEHcebE4XmHY2b7wAnGSsotDzzLpu2NLoNs1gM4wVjJWPrKRv7tyRf5+EmHcfTwA/IOx8z2UaYJRtJZkpZJWi7p6x20u1BSSKpJlysk3SFpgaQlkq4u1KekxyTNS39elnR3lmOzzhURXDdzMZV9K/jiGS6DbNYTZFYyWVI5SenjDwB1wFxJMyNi8W7tKoErgadarb4I6BMRkyT1BxZL+hWwqr0+I+K9rfr8LfC7rMZmne/+Ra/wxIrXuf6CiRw4oHfe4ZhZJ8jyCGYysDwiVkREA3AXcEEb7W4AbgS2t1oXwABJvYB+QAOwsZg+04Q1BfARTDexvbGZ78xewlGHVHLJ5NF5h2NmnSTLBDOS5Ihjp7p03S6SjgdGRcSs3badAWwB1gAvATdFxBvF9An8NfBgRGzc5xFYl/jFYyuoW7+Na86fQC+XQTbrMTI7RQa0dQtQ7HpTKgNuAS5to91koBkYARwIPCbpgUJ9pi4GftFuUNLlwOUAo0f723LeXqnfzq0PP8+ZEw/h5LFD8w7HzDpRll8X64BRrZargJdbLVcCxwCPSFoJnATMTC/0XwLcFxGNEbEWeByoKdSnpINIktPs9oKKiNsioiYiaoYNG7YPw7PO8P17l9AcwTfOcRlks54mywQzFzhS0hhJvYFpwMydb0ZEfUQMjYjqiKgGngSmRkQtyWmxKUoMIEk+Swv1SXJzwKyIaH09x0rU0y++wd3zXuZ/vncMow/qn3c4ZtbJMkswEdEEXAHcDywBfhMRiyRdL2lqgc1vBQYCC0mSyvSImN9en622mwb8qpOHYhloaQmuu2cxhxzQh/91ussgm/VEWV6DISLmAHN2W3dNO21Pb/V6M8nRSFF9ttWHlbYZf6ljfl09t3z0OAb0yfTX0Mxy4lt2rMtt2t7Ijfct44TRg/mrd+x+E6CZ9RT+6mhd7scPLee1zTv450/WeL4xsx7MRzDWpVas28ztj7/ARSdWcdyowXmHY2YZcoKxLvXd2Uvo06ucr5x1VN6hmFnGnGCsyzyybC0PLl3L304Zy8GVLoNs1tM5wViXaGxu4fpZi6k+qD+Xnlyddzhm1gWcYKxL3PH/VrJi3Ra+dd4E+vRyGWSz/YETjGXutc07+OGDz3HquGFMOfrgvMMxsy7iBGOZ+8Hvl7GtoZlrzhvv25LN9iNOMJaphavruWvuKv7Hu6sZe3Bl3uGYWRdygrHMRATX3bOIA/v35n+fcWTe4ZhZF3OCsczMmr+GuSvXc9UHj2JQv4q8wzGzLuYEY5nY1tDM9+YsYcKhB/DRd44qvIGZ9ThOMJaJn/7xeV6u3863p06kvMwX9s32R04w1ulWb9jGT//4POcdeyiTxwzJOxwzy0lRCUbSyWllSSR9XNLNkg7LNjTrrv7PnCVIcPU54/MOxcxyVOwRzD8BWyUdB3wVeBG4s9BGks6StEzScklf76DdhZJCUk26XCHpDkkLJC2RdHWhPtPyyt+V9Gy6zZVFjs060VMrXmf2/DV87rQjGDm4X97hmFmOik0wTRERwAXADyPih0CHDzVIKicpfXw2MAG4WNKENtpVAlcCT7VafRHQJyImAScCn5VUXaDPS4FRwNERMR64q8ixWSdpbgm+fc9iRgzqy2dPPSLvcMwsZ8UmmE3pUcTHgdnpjr7QfaeTgeURsSIiGkh2+Be00e4G4EZge6t1AQyQ1AvoBzQAGwv0+Xng+ohoAYiItUWOzTrJr+euYsmajfzduePp19vzjZnt74pNMB8FdgCfjohXgJHAPxTYZiSwqtVyXbpuF0nHA6MiYtZu284AtgBrgJeAmyLijQJ9HgF8VFKtpHsl+cm+LlS/tZGbfr+MyWOGcO6kQ/MOx8xKQLElkzeRnBprljQOOBr4VYFt2ro3NXa9KZUBt5Cc2trdZKAZGAEcCDwm6YECffYBtkdEjaQPAbcD731bUNLlwOUAo0ePLjAEK9YPH3yO9VsbuPb8CZ5vzMyA4o9gHgX6SBoJPAh8CviXAtvUkVwT2akKeLnVciVwDPCIpJXAScDM9EL/JcB9EdGYnup6HKgp0Gcd8Nv09X8Bx7YVVETcFhE1EVEzbNiwAkOwYixfu4k7n1jJtHeOZuKIQXmHY2YlotgEo4jYCnwI+FFE/DUwscA2c4EjJY2R1BuYBszc+WZE1EfE0Iiojohq4ElgakTUkpwWm5LeGTaAJPksLdDn3cCU9PVpwLNFjs32QTLf2GL69S7nqg+OyzscMyshRScYSe8GPgbMTtd1eBU3IpqAK4D7gSXAbyJikaTrJU0t8PfdCgwEFpIklekRMb+9PtNtvg98WNIC4HvAZ4ocm+2DB5es5bHnXuMLZ4zjoIF98g7HzEqIkruPCzSSTgO+DDweEX8v6XDgCxHRrZ81qampidra2rzD6LZ2NDVz5i2PUl4m7vvCqVSUe2IIs/2BpKcjoqZQu6Iu8kfEH4E/SqqUNDAiVpA8u2L7semPr2Tl61u547LJTi5m9jbFThUzSdJ/k5yyWizpaUmFrsFYD7Z203Z+9OBznDH+YE4b55slzOztiv3a+TPgSxFxWESMJjld9vPswrJSd+N9y2hobuEb575tcgYzM6D4BDMgIh7euRARjwADMonISt4zqzYw4+k6LjtlDGOG+tfAzNpW7IOWKyR9C/jXdPnjwAvZhGSlrKUl+PY9ixg6sA9XvG9s3uGYWQkr9gjmMmAY8J8kDzEOI3nY0vYzv3tmNf/90ga+dtZRVPZ1GWQza1+xd5Gtx3eN7fe27Gji+/cu5biqQXz4hKq8wzGzEtdhgpF0D63mD9tdRBR6YNJ6kJ88spxXN+7gJx87kTKXQTazAgodwdzUJVFYyXvp9a38/LEX+OvjR3LiYQfmHY6ZdQMdJpj0AUszvjtnMb3KxNfOOjrvUMysmyjqGkw6v9fup8rqgVrgOxHxemcHZqXj8eWvcf+iV/nKmUcxfFDfvMMxs26i2NuU7yWpz/Lv6fI0ktos9STT9p/f6ZFZSWhqbuG6exYxakg/Pn3KmLzDMbNupNgEc3JEnNxqeYGkxyPiZEkfzyIwKw2/fOolnn11Mz/9+In0rXAZZDMrXrHPwQyU9K6dC5Imk0ynD9DU6VFZSVi/pYGb//AsJ489iDMnHpJ3OGbWzRR7BPMZ4HZJA0lOjW0EPp0WA/teVsFZvm7+w7Ns2t7INedNdBlkM9tjxT5oOReYJGkQSQ2ZDa3e/k0mkVmulr6ykV8+9SIfP+kwjhpemXc4ZtYNFTtd/yBJNwMPAg9I+kGabKwHigium7mYA/pV8KUPuAyyme2dYq/B3A5sAj6S/mwEphfaSNJZkpZJWi7p6x20u1BSSKpJlysk3SFpgaQlkq4u1Kekf5H0gqR56c87ihyb7eb+Ra/wxIrX+dIHxjG4f++8wzGzbqrYazBHRMSHWy1fJ2leRxtIKgduBT4A1AFzJc2MiMW7taskmefsqVarLwL6RMQkSf1Jipz9ClhVoM+vRMSMIsdkbdje2Mx3Zi/hqEMquWTy6LzDMbNurNgjmG2STtm5IOlkYFuBbSYDyyNiRUQ0AHcBF7TR7gbgRmB7q3UBDJDUC+gHNJAcNRXbp+2lXzy2grr127j2/An0chlkM9sHxe5BPg/cKmmlpBeBHwOfK7DNSJIjjp3q0nW7SDoeGBURs3bbdgawBVgDvATcFBFvFNHndyXNl3SLpD5tBSXpckm1kmrXrVtXYAj7lzX127j14ec5a+Jw3jN2aN7hmFk3V1SCiYh5EXEccCwwKSKOj4hnCmzW1n2tu6abkVQG3EJSfnl3k0lmDhgBjAG+LOnwAn1eDRwNvBMYAnytnbHcFhE1EVEzbJhrybf29/cupTmCb5w7Pu9QzKwHKDRd/5faWQ9ARNzcweZ1wKhWy1XAy62WK4FjgEfS/oYDMyVNBS4B7ouIRmCtpMeBGpKjlzb7jIg16bodkqYDV3U0Nnurp198g7vnvcwV7xvLqCH98w7HzHqAQkcwlQV+OjIXOFLSGEm9SeYvm7nzzYioj4ihEVEdEdXAk8DUiKglOS02RYkBwEnA0o76lHRo+qeAvwIWFvlvsN9raQm+PXMxhxzQh8+ffkTe4ZhZD1Fouv7r9rbjiGiSdAVwP1AO3B4RiyRdD9RGxMwONr+V5DbohSSnxaZHxHyAtvpMt/mlpGFp+3kUvkZkqRlP17FgdT3/+NF3MKBPsTcWmpl1TBHtFqxsewPpLxFxQkbxdKmampqora3NO4xcbdreyPtueoTRQ/rz28+/x1PCmFlBkp6OiJpC7fbm66r3QD3Ijx5azmubG7j90nc6uZhZp9qbBx1md3oUlosV6zYz/fEXuOjEKo6tGpx3OGbWw+xxgomIb2YRiHW978xeQp9e5XzlrKPyDsXMeqBiJ7vcJGnjbj+rJP1X+nyKdTMPL1vLQ0vX8rdTxnJwpcsgm1nnK/YazM0kz5v8O8k1mGkkz60sI5kI8/QsgrNsNDS1cMOsxYwZOoBPnewyyGaWjWJPkZ0VET+LiE0RsTEibgPOiYhfAwdmGJ9l4M4nVrJi3Ra+ee54evfyfGNmlo1i9y4tkj4iqSz9+Uir9/bsPmfL1Wubd/DDB57jtHHDmHL0wXmHY2Y9WLEJ5mPAJ4C1wKvp649L6gdckVFsloEf/H4Z2xqb+dZ5E3xbspllqtiSySuA89t5+0+dF45laeHqeu6au4rLTh7D2IMH5h2OmfVwxd5FNk7Sg5IWpsvHSvLtyt1IRHDdPYsY0r83V77/yLzDMbP9QLGnyH5OMh1+I0A6L9i0rIKyzjdr/hrmrlzPVWcexaB+FXmHY2b7gWITTP+I+PNu65o6OxjLxraGZr43ZwkTRxzAR2pGFd7AzKwTFJtgXpN0BOkdY5IuJKk2ad3AT//4PC/Xb+fa8ydSXuYL+2bWNYp90PJvgNuAoyWtBl4gubPMSlzd+q389I/Pc96xhzJ5zJC8wzGz/UixCWY1SX2Wh0nKEW8EPglcn1Fc1km+d+9SJLj6HJdBNrOuVWyC+R2wAfgLby17bCXsyRWvM3v+Gr5wxpGMHNwv73DMbD9T7DWYqoiYFhE3RsQPdv4U2kjSWZKWSVou6esdtLtQUkiqSZcrJN0haYGkJZKuLrZPST+StLnIcfVYzS3BdfcsZuTgfnz2VJdBNrOuV2yC+X+SJu1Jx5LKSUofnw1MAC6WNKGNdpXAlcBTrVZfBPSJiEnAicBnJVUX6jNNUC5sAtw19yWWrNnI1eccTb/e5XmHY2b7oWITzCnA0+mRw/z0yGJ+gW0mA8sjYkVENAB3ARe00e4G4EZge6t1AQyQ1AvoBzSQXPdpt880+fwD8NUix9Rj1W9t5Kb7lzF5zBDOnXRo3uGY2X6q2GswZ+9F3yOBVa2W64B3tW4g6XhgVETMknRVq7dmkCSONUB/4IsR8Yakjvq8ApgZEWs6mmNL0uXA5QCjR4/ei2GVvn988FnqtzVy7fmeb8zM8lPsXGQv7kXfbe3Zds28LKkMuAW4tI12k4FmYARJOYDHJD3QXp+SRpCcVju9UFBpqYHbAGpqanrcTNDPvbqJO594kWmTRzNxxKC8wzGz/VixRzB7ow5o/dh4FW+9A60SOAZ4JP2WPRyYKWkqcAlwX0Q0AmslPQ7UkBy9tNXn8cBYYHnaV39JyyNibBYDK1URwfWzFtO/dzlf/sC4vMMxs/1cltWm5gJHShojqTfJ3GUzd74ZEfURMTQiqiOiGngSmBoRtcBLwBQlBgAnAUvb6zMiZkfE8FZ9bd3fkgvAg0vW8thzr/HFM8Zx0MA+eYdjZvu5zBJMRDSRXBe5H1gC/CYiFkm6Pj1K6citwEBgIUlSmR4R89vrM6sxdCc7mpq5YfZixh48kE+8+7C8wzEzy/QUGRExB5iz27pr2ml7eqvXm0muqRTVZxtt9rtiJ9MfX8mLr2/ljssmU1HuMshmlj/viXqAtRu386MHn+OM8Qdz2rhheYdjZgY4wfQIN96/jIbmFr557tueYzUzy40TTDc3b9UGZjxdx2WnjKF66IC8wzEz28UJphtraQm+PXMRwyr78LdTXAbZzEqLE0w3dve81cxbtYGvnnkUA/tker+Gmdkec4LpprbsaOL79y7luKpBfPiEqrzDMTN7GyeYburWh5ezdtMOrp06kTKXQTazEuQE0w299PpWfvHYC3zo+JGcMPrAvMMxM2uTE0w39J3Zi+lVLr529tF5h2Jm1i4nmG7mT8+9xu8Xv8rfvG8shxzQN+9wzMza5QTTjTQ1t3D9rEWMHtKfT58yJu9wzMw65ATTjfzyqZd49tXNfOPc8fStcBlkMyttTjDdxPotDdz8h2c5eexBfHDCIXmHY2ZWkBNMN3HzH55l844mrjlvossgm1m34ATTDSxZs5FfPvUiH3/XaI4aXpl3OGZmRXGCKXERwfX3LOaAfhV80WWQzawbyTTBSDpL0jJJyyV9vYN2F0oKSTXpcoWkOyQtkLRE0tWF+pT0z5KekTRf0gxJPaLo2H0LX+GJFa/z5Q+MY3D/3nmHY2ZWtMwSjKRyktLHZwMTgIslva1giaRK4ErgqVarLwL6RMQk4ETgs5KqC/T5xYg4LiKOBV4iKa3crW1vbOa7c5Zw9PBKLp48Ou9wzMz2SJZHMJOB5RGxIiIagLuAC9podwNwI7C91boABkjqBfQDGoCNHfUZERsBlFwB75f20a39/NEV1K3fxjXnT6CXyyCbWTeT5V5rJLCq1XJdum4XSccDoyJi1m7bzgC2AGtIjkZuiog3CvUpaTrwCnA08KPOGUY+1tRv4yePPM/ZxwznPUcMzTscM7M9lmWCaete2l1HFZLKgFuAL7fRbjLQDIwAxgBflnR4oT4j4lPpNkuAj7YZlHS5pFpJtevWrStyKF3v+/cupTmCvztnfN6hmJntlSwTTB0wqtVyFfByq+VK4BjgEUkrgZOAmemF/kuA+yKiMSLWAo8DNUX0SUQ0A78GPtxWUBFxW0TURETNsGHD9mF42ald+Qa/m/cynz31cEYN6Z93OGZmeyXLBDMXOFLSGEm9gWnAzJ1vRkR9RAyNiOqIqAaeBKZGRC3JabEpSgwgST5L2+szbTcWdl2DOT9t3+20tATX3bOY4Qf05fOnH5F3OGZmey2zOrsR0STpCuB+oBy4PSIWSboeqI2ImR1sfiswHVhIclpsekTMB2inzzLgDkkHpO37Vu4/AAAM7UlEQVSfAT6f1diyNOPpOhasrueH095B/94ug2xm3Zciuv3NVnutpqYmamtr8w5jl43bG5ly0yMcdtAAZnzu3Z4SxsxKkqSnI6KmUDt/RS4hP35oOa9vaWD6pZOdXMys2/PDFSVixbrNTH/8BS46sYpJVYPyDsfMbJ85wZSI78xeQp9e5Vx15lF5h2Jm1imcYErAw8vW8tDStVz5/rEcXOkyyGbWMzjB5KyhqYUbZi1mzNABXPoel0E2s57DCSZndz6xkhXrtvCt88bTu5c/DjPrObxHy9Frm3fwwwee47Rxw3jfUQfnHY6ZWadygsnRD36/jG2NzXzrvAm+LdnMehwnmJwsXF3PXXNX8cn3VDP24B5RG83M7C2cYHIQEVx3zyKG9O/Nle8/Mu9wzMwy4QSTg3vmr2HuyvVcdeZRDOpXkXc4ZmaZcILpYtsamvnenCVMHHEAH6kZVXgDM7Nuygmmi/3TH59nTf12rj1/IuVlvrBvZj2XE0wXqlu/lZ/98XnOP24Ek8cMyTscM7NMOcF0oe/NWYoEV599dN6hmJllzgmmizy54nVmL1jD508by4jB/fIOx8wsc5kmGElnSVomabmkr3fQ7kJJIakmXa6QdIekBZKWSLq6UJ+SfpmuXyjpdkklc3tWc1oGeeTgflx+6uF5h2Nm1iUySzCSyklKH58NTAAuljShjXaVwJXAU61WXwT0iYhJwInAZyVVF+jzl8DRwCSgH/CZTAa2F+6a+xJL1mzk784ZT7/e5XmHY2bWJbI8gpkMLI+IFRHRANwFXNBGuxuAG4HtrdYFMEBSL5Jk0QBs7KjPiJgTKeDPQFVG49oj9Vsbuen+ZbxrzBDOmTQ873DMzLpMlglmJLCq1XJdum4XSccDoyJi1m7bzgC2AGuAl4CbIuKNIvusAD4B3NcJY9hn//jgs9Rva+Sa8z3fmJntX3pl2Hdbe9PY9aZUBtwCXNpGu8lAMzACOBB4TNIDhfpM/QR4NCIeazMo6XLgcoDRo0d3PIJ99Nyrm7jziReZNnk0E0e4DLKZ7V+yPIKpA1o/ql4FvNxquRI4BnhE0krgJGBmeqH/EuC+iGiMiLXA40BNoT4lXQsMA77UXlARcVtE1EREzbBhw/ZheB2LCK6ftZgBvcv58gfGZfb3mJmVqiwTzFzgSEljJPUGpgEzd74ZEfURMTQiqiOiGngSmBoRtSSnxaYoMYAk+SztqE9JnwHOBC6OiJYMx1WUB5as5bHnXuMLZ4zjoIF98g7HzKzLZZZgIqIJuAK4H1gC/CYiFkm6XtLUApvfCgwEFpIklekRMb+9PtNtfgocAjwhaZ6kazp/VMXZ0dTMd2YvZuzBA/nEuw/LKwwzs1xleQ2GiJgDzNltXZs7/og4vdXrzSS3KhfVZ7o+07Hsidv/tJIXX9/KnZdNpqLcz7Ka2f7Je79Otnbjdn780HOcMf4QTh2X3TUeM7NS5wTTyf7+vmU0NgffPHd83qGYmeXKCaYTzVu1gd/+pY7LThlD9dABeYdjZpYrJ5hO0tISfHvmIoZV9uGKKWPzDsfMLHdOMJ3k7nmrmbdqA18762gG9imZ+w3MzHLjBNMJNu9o4vv3LuW4UYP50PEjC29gZrYfcILpBD95eDlrN+3g2vMnUOYyyGZmgBPMPnvx9S384rEX+NDxIzlh9IF5h2NmVjKcYPbRd2cvoVe5+JrLIJuZvYUTzD7403Ov8fvFr/I37xvLIQf0zTscM7OS4gSzl5qaW7junkWMHtKfT58yJu9wzMxKjhPMXvq3J1/kubWb+ca54+lb4TLIZma7c4LZC29saeDmPzzLKWOH8sEJh+QdjplZSXKC2Qs3/2EZWxqaXQbZzKwDTjB7YfSQ/lx+6uGMO6Qy71DMzEqW5zTZC5efekTeIZiZlbxMj2AknSVpmaTlkr7eQbsLJYWkmnS5QtIdkhZIWiLp6kJ9SroiXReShmY5LjMzKyyzBCOpnKT08dnABOBiSRPaaFcJXAk81Wr1RUCfiJgEnAh8VlJ1gT4fB84AXsxoSGZmtgeyPIKZDCyPiBUR0QDcBVzQRrsbgBuB7a3WBTBAUi+gH9AAbOyoz4j474hYmdVgzMxsz2SZYEYCq1ot16XrdpF0PDAqImbttu0MYAuwBngJuCki3iimTzMzKw1ZXuRv6/7d2PWmVAbcAlzaRrvJQDMwAjgQeEzSA4X6LCoo6XLgcoDRo0fvyaZmZrYHsjyCqQNGtVquAl5utVwJHAM8ImklcBIwM73QfwlwX0Q0RsRakusrNUX0WVBE3BYRNRFRM2zYsD0ckpmZFSvLBDMXOFLSGEm9gWnAzJ1vRkR9RAyNiOqIqAaeBKZGRC3JabEpSgwgST5LC/VpZmalI7MEExFNwBXA/cAS4DcRsUjS9ZKmFtj8VmAgsJAkqUyPiPnt9Qkg6UpJdSRHNfMl/SKTgZmZWVEUsUeXMHoUSevY+9uahwKvdWI4eeopY+kp4wCPpVT1lLHs6zgOi4iC1xj26wSzLyTVRkRN3nF0hp4ylp4yDvBYSlVPGUtXjcNzkZmZWSacYMzMLBNOMHvvtrwD6EQ9ZSw9ZRzgsZSqnjKWLhmHr8GYmVkmfARjZmaZcIIpoFDJAUl9JP06ff8pSdVdH2VhRYzjUknrJM1Lfz6TR5zFkHS7pLWSFrbzviT933Ss8yWd0NUxFqOIcZwuqb7VZ3JNV8dYLEmjJD2cltdYJOl/t9Gm5D+XIsfRLT4XSX0l/VnSM+lYrmujTbb7r4jwTzs/QDnwPHA40Bt4BpiwW5v/Bfw0fT0N+HXece/lOC4Ffpx3rEWO51TgBGBhO++fA9xLMnfdScBTece8l+M4HZiVd5xFjuVQ4IT0dSXwbBu/YyX/uRQ5jm7xuaT/zgPT1xUkJVFO2q1NpvsvH8F0rJiSAxcAd6SvZwDvl9TWpJx5KrZ0QrcQEY8Cb3TQ5ALgzkg8CQyWdGjXRFe8IsbRbUTEmoj4S/p6E8lMG7vPdF7yn0uR4+gW0n/nzeliRfqz+0X3TPdfTjAdK6Y8wK42kUxlUw8c1CXRFa/YMgcfTk9dzJA0qo33u4ueVNbh3ekpjnslTcw7mGKkp1mO561FBKGbfS4djAO6yeciqVzSPGAt8IeIaPczyWL/5QTTsWLKA+xzCYEuUEyM9wDVEXEs8ABvfqvpjrrDZ1KMv5BMyXEc8CPg7pzjKUjSQOC3wBciYuPub7exSUl+LgXG0W0+l4hojoh3kMzROFnSMbs1yfQzcYLpWDHlAXa1SStwDqL0TnsUHEdEvB4RO9LFn5OUqu6u9rmsQymIiI07T3FExBygQtLQnMNql6QKkp3yLyPiP9to0i0+l0Lj6G6fC0BEbAAeAc7a7a1M919OMB0rpjzATOCT6esLgYcivWJWQgqOY7dz4VNJzj13VzOB/5HetXQSUB8Ra/IOak9JGr7zfLikyST/X1/PN6q2pXH+M7AkIm5up1nJfy7FjKO7fC6ShkkanL7uB5xBUvaktUz3X1lWtOz2IqJJ0s7yAOXA7ZGWHABqI2ImyS/jv0paTpL5p+UXcduKHMeVSsooNJGM49LcAi5A0q9I7uQZqqREw7UkFzCJiJ8Cc0juWFoObAU+lU+kHStiHBcCn5fUBGwDppXgl5edTgY+ASxIz/kD/B0wGrrV51LMOLrL53IocIekcpIk+JuImNWV+y8/yW9mZpnwKTIzM8uEE4yZmWXCCcbMzDLhBGNmZplwgjEzs0w4wZhlSNLmwq063H6GpMPT1wMl/UzS8+nsuI9Kepek3ulrP3ZgJcUJxqxEpXNclUfEinTVL0ieVTgyIiaSPKs0NJ3A9EHgo7kEatYOJxizLpA+vf4PkhZKWiDpo+n6Mkk/SY9IZkmaI+nCdLOPAb9L2x0BvAv4ZkS0AKSzY89O296dtjcrGT6kNusaHwLeARwHDAXmSnqU5MnxamAScDDJFD23p9ucDPwqfT0RmBcRze30vxB4ZyaRm+0lH8GYdY1TgF+ls9u+CvyRJCGcAvxHRLRExCvAw622ORRYV0znaeJpkFTZyXGb7TUnGLOu0V4Rp46KO20D+qavFwHHSero/2wfYPtexGaWCScYs67xKPDRtADUMJJyyX8G/kRS6K1M0iEkk1/utAQYCxARzwO1wHWtZvI9UtIF6euDgHUR0dhVAzIrxAnGrGv8FzAfeAZ4CPhqekrstyQ1ORYCPyOpnlifbjObtyaczwDDgeWSFpDU7dlZT+V9JLMVm5UMz6ZsljNJAyNic3oU8mfg5Ih4Ja3h8XC63N7F/Z19/CdwdUQs64KQzYriu8jM8jcrLQzVG7ghPbIhIrZJupakbvpL7W2cFpG728nFSo2PYMzMLBO+BmNmZplwgjEzs0w4wZiZWSacYMzMLBNOMGZmlgknGDMzy8T/B0qUN6knLBZcAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dict with classes as the keys, and the values as the grid of scores obtained during cross-validating each fold,\n",
    "# Each dict value has shape (n_folds, len(Cs))\n",
    "n_Cs = len(Cs)    \n",
    "mse_mean = -np.mean(lr_cv_L2.scores_[1], axis=0)\n",
    "pyplot.plot(np.log10(Cs), mse_mean.reshape(n_Cs,1)) \n",
    "#plt.plot(np.log10(reg.Cs)*np.ones(3), [0.28, 0.29, 0.30])\n",
    "pyplot.xlabel('log(C)')\n",
    "pyplot.ylabel('neg-logloss')\n",
    "pyplot.show()\n",
    "\n",
    "#print ('C is:',lr_cv.C_)  #对多类分类问题，每个类别的分类器有一个C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "log(C) = 0时，即C = 1时，Jcv(cross validation error)最小"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测\n",
    "评价带L2正则项的LogisticRegressionCV预测性能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the correct classification fraction of the zero_one_loss of LogisticRegressionCV on test is 0.7337662337662337\n",
      "the correct classification fraction of the zero_one_loss of LogisticRegressionCV on train is 0.7833876221498371\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import zero_one_loss  #评价逻辑回归预测模型的性能\n",
    "# 预测\n",
    "y_test_pred_lr = lr_cv_L2.predict(X_test)\n",
    "y_train_pred_lr = lr_cv_L2.predict(X_train)\n",
    "# 使用zero_one_loss评价模型在测试集和训练集上的性能，并输出评估结果\n",
    "#测试集\n",
    "print ('the correct classification fraction of the zero_one_loss of LogisticRegressionCV on test is', 1 - zero_one_loss(y_test, y_test_pred_lr))\n",
    "#训练集\n",
    "print ('the correct classification fraction of the zero_one_loss of LogisticRegressionCV on train is', 1 - zero_one_loss(y_train, y_train_pred_lr))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用带L2正则项的LogisticRegressionCV做预测，测试集上正确分类的比例是0.7337662337662337，训练集上正确分类的比例是0.7833876221498371"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    ""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3.0
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}